Mine dust and fog image defogging method and system based on three-dimensional inverse interaction intersection

Abstract

A method and system for dehazing mine dust and fog images based on 3D inverse interactive cross-connection is proposed. The method involves adding a 3D attention extraction module to the u-net network architecture, and adding spatially regularized inverse cross-attention and inverse interactive connection modules to the downstream tasks. The 3D attention extraction module enhances the model's ability to perceive non-uniform fog and haze. The spatially regularized inverse cross-attention module dynamically and adaptively highlights residual fog, artifacts, and contrast loss areas, and strengthens the depth estimation network's response to these areas, improving the overall performance and robustness of both tasks. The inverse interactive connection module improves the collaborative efficiency of the dehazing and depth estimation networks and also solves the problem of error information not being fed back to the feature layer. The system adopts an encoder-decoder architecture, which includes a 3D attention extraction module, an inverse cross-regularized attention module, and an inverse interactive connection module. This invention improves the accuracy of monitoring camera systems and intelligent detection systems in mines.

Description

Technical Field

[0001] This invention relates to a method and system for dehazing mine dust and fog images based on three-dimensional inverse interactive cross-section, belonging to the field of image dehazing technology. Background Technology

[0002] In the underground environment of coal mines, coal ash and dust generated during long-term operations diffuse into the air, and the widespread use of spray dust suppression systems in mine work areas results in high concentrations of dust and fog in the air. In addition, complex lighting conditions exist inside the mine, including miners' lamps, LED lighting, and reflective wall surfaces. These factors lead to serious degradation of the image quality captured by imaging equipment, specifically manifested as: (1) low contrast: coal dust absorbs light, resulting in an overall dark image and blurred target edges; (2) local overexposure and halo phenomenon: strong light sources such as miners' lamps illuminate a foggy environment, forming a halo phenomenon and increasing the degree of image blur; (3) local dense fog residue: the fog droplets in the spray dust suppression area are larger, enhancing light scattering and significantly reducing visibility in local areas. These problems directly affect the operation of the monitoring camera system and intelligent detection system in the mine, seriously interfering with the accuracy of tasks such as target detection, personnel identification, and equipment monitoring.

[0003] To address the above issues, recent depth estimation tasks have focused on enhancing dehazing modeling; however, the structural information of the residual regions remains largely unutilized, limiting task collaboration. The Depth Information Assisted Collaborative Mutual Promotion Network (DEM) joint training framework proposed at CVPR 2024 has improved performance to some extent, but it still relies on static weighting and differencing mechanisms, lacking 3D structural modeling and feature-level interaction, thus restricting deep fusion between tasks. In summary, current two-stage task models still have significant shortcomings in feature interaction, adaptability, and spatial structure modeling, making it difficult to achieve closer collaborative optimization and performance improvement between image dehazing and depth estimation tasks. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for dehazing mine dust and fog images based on three-dimensional inverse interaction. This method and system can achieve close collaborative optimization and performance improvement between image dehazing and depth estimation tasks, improve the accuracy of monitoring camera systems and intelligent detection systems in mines, and improve the level of coal mine safety production.

[0005] To achieve the above objectives, the present invention provides a method for dehazing mine dust and fog images based on three-dimensional inverse interactive cross-section, comprising the following steps:

[0006] S1. Add a 3D attention extraction module to the u-net network architecture, and add a spatially regularized inverse cross-attention module and an inverse interactive connection module to the downstream task.

[0007] S2. In the upstream task, 3D attention is extracted from the fusion tensor of the foggy depth map obtained by the depth estimation network and the preliminary feature map obtained by the u-net network. The form of dilated 3D convolution + coordinate attention is used to further extract 3D information.

[0008] S3. In the downstream task, on the one hand, the dehazed image output by the u-net network and the real image are subjected to spatial regularization inverse cross attention, and the regions with unsatisfactory dehazing are highlighted through inverse numerical processing. At the same time, a spatial regularization module is added to highlight the relative positional relationship between the regions with unsatisfactory dehazing. The spatial regularization loss function in the depth estimation network guides the depth estimation network to focus on the regions with unsatisfactory residual fog, thereby strengthening the mutual learning between the networks.

[0009] On the other hand, the dehazed image is processed by a depth estimation network to obtain a dehazed depth map. This map is then subjected to spatial regularization inverse cross-attention with the real foggy depth map to highlight the regions with poor depth estimation and their relative positional relationships. The spatial regularization loss function of the dehazed network is used to guide the optimization of the network, making it more focused on learning the spatial structure of the regions with poor depth estimation and thus promoting the two-stage task.

[0010] S4. Input the obtained regions with global positional relationships and regions with poor depth estimation of the dehazed image into the inverse interactive connection module. Through the constraint of regularized similarity loss function, the regions with poor dehazed image and regions with poor depth estimation learn each other's spatial structure, and at the same time optimize the mutual network, namely the depth estimation network and the dehazed network.

[0011] Furthermore, the areas with unsatisfactory dehazing refer to areas with obvious fog residue, insufficient detail recovery, blurred edges, color distortion, or localized reduced contrast, and where the peak signal-to-noise ratio (PSNR) is below 25 dB or the structural similarity index (SSIM) is below 0.9; conversely, areas with ideal dehazing are those that meet these criteria. Similarly, areas with unsatisfactory dehazing image depth refer to areas where the depth value deviates from the true value, the depth boundary is blurred, local depth continuity is disrupted, or abnormal jumps occur; conversely, areas with ideal dehazing image depth are those that meet these criteria.

[0012] Furthermore, the u-net network architecture in S1 includes three encoders and two decoders. Each encoder and decoder includes a ResBlock block, and a three-dimensional attention extraction module is added after each ResBlock block. The first attention module is used to receive the depth information of the feature map, and before the first ResBlock block is the fusion tensor of the depth map and the feature map, as well as the multi-scale feature extraction module u-net. Through the extraction of multi-scale three-dimensional attention, the network extracts depth and spatial features.

[0013] Furthermore, the 3D attention extraction module described in S2 is an adaptive multi-scale 3D attention module, which includes a channel fusion layer, a convolutional layer, a normalization layer, a linear layer, a 3D dilated convolution, and coordinate attention units; the working process of the 3D attention extraction module is as follows:

[0014] S2.1 After merging via Concat channels, the output is denoted as F1:

[0015] F1 = Concat(Depth1 + u1 + u2);

[0016] Wherein, Concat is the channel fusion layer, Depth1 is the output image of the foggy image after passing through the depth estimation network, u1 is the feature map extracted by the multi-scale feature extraction module u-net network, u2 is the original color image; + indicates channel stacking;

[0017] S2.2 Initial feature extraction is performed using the Conv2d layer, with the following expression:

[0018] F2 = σ(W1*x + b1);

[0019] Where X is the input image, W1 and b1 are two-dimensional convolution kernel parameters, * represents the convolution operation, and σ is a non-linear activation function;

[0020] S2.3. Normalization layer norms accelerate model convergence and stability, as shown in the following expression:

[0021]

[0022] Where μ and δ are the mean and standard deviation calculated for the current batch, respectively, and γ and β are learnable parameters.

[0023] S2.4 The network learning extracts features at different levels through three parallel linear layer branches. The outputs of the different linear layers are A1, A2, and A3, as shown in the following expressions:

[0024] A1 = W2 * F' + b2;

[0025] A2 = W3 * F' + b3;

[0026] A3 = W4 * F' + b4;

[0027] Where W2, W3, W4 and b2, b3, b4 are the weights and biases of A1, A2, and A3, respectively;

[0028] S2.5. Further extract depth and spatial feature information through 3D convolutional layers. The outputs of different 3D dilated convolutional layers are C1, C2, and C3, respectively, and their expressions are as follows:

[0029] C1 = σ(W5*A1 + b5);

[0030] C2 = σ(W6*A2 + b6);

[0031] C3 = σ(W7*A3 + b7);

[0032] Where W5, W6, and W7 refer to convolutional kernels with different dilation rates; b5, b6, and b7 are the biases of convolutional layers with different dilation rates; and σ is a nonlinear activation function.

[0033] After channel fusion, feature mappings for different branches are obtained:

[0034] F out =Concat(C1+C2+C3);

[0035] S2.6, the coordinate attention unit includes horizontal and vertical global average pooling layers, spatial fusion layers, ordinary convolutional layers, batch normalization layers, and nonlinear layers; among them, the expression for the working process of the horizontal and vertical global average pooling layers is:

[0036]

[0037] Where w and H represent the width and height of the feature map, respectively; i and j represent the row and column indices, respectively;

[0038] S2.7. Perform matrix transpose on 1*W*C and then concatenate according to spatial order. The expression is:

[0039] f=σ(F2([Z h Z w ]));

[0040] Here, F2 refers to the spatial concatenation and fusion of the horizontal and vertical weight matrices, and Z... h Z w This refers to the transpose of the weight matrix obtained from horizontal and vertical global average pooling; σ refers to the non-linear activation function.

[0041] S2.8. To accelerate training, normalization and non-linear activation are used. The data is further divided into H*1*C and W*1*C, and sigmoid() is applied to generate two attention vectors, g and g, respectively. h and g w Among them, g h ∈C*1*W、g w ∈C*1*h, the expression is as follows:

[0042] g h =σ(F h (f h ));

[0043] g w =σ(F w (f w ));

[0044] F h (f h This refers to further dividing the fusion weight matrix f obtained above into H*1*C feature column vectors in the form of column vectors; F w (f w This refers to further dividing the fusion weight matrix f obtained above into W*1*C feature row vectors in the form of row vectors; σ is a non-linear activation function;

[0045] The output is:

[0046] Where i and j represent the row index and column index, respectively; x c (i,j) refers to the feature map of the original input. This refers to the attention weights of the column vectors obtained by applying coordinate attention to the original feature map. This refers to the row vector attention weights obtained by applying coordinate attention to the original feature map;

[0047] Furthermore, the spatially regularized inverse cross-attention module in S3 includes a cross-attention unit, a numerical processing unit, and a spatial regularization unit, and its operation process is as follows:

[0048] S3.1, D I G represents the image after dehazing. T This represents the corresponding real-world haze-free image. In a deep learning framework, the dehazed image D... I and true fog-free images G T After passing through the feature extraction network, the corresponding feature representations are obtained respectively:

[0049] F d =F(D I );

[0050] F c=F(G T );

[0051] Where F(.) represents the feature extraction network;

[0052] S3.2. Employ the Cross-Attention mechanism to calculate F. d and F c The attention weights between them are expressed as follows:

[0053] Q = F d W q ;

[0054] K = F c W k ;

[0055] V = F c W V ;

[0056] Where Q represents the query, K represents the key, V represents the value, and W represents the value. q W k W V All are linear transformation matrices. S3.3 and cross-attention are calculated as follows:

[0057]

[0058] Where A is the cross-attention matrix, with a size of H×W, representing the similarity between each region of the dehazed image and the real haze-free image; d k is the scaling factor of the key feature dimension, used to stabilize the gradient; since A reflects the degree of matching between the dehazed image and the haze-free image, its high-weight region, that is, the region with a weight value greater than 0.7 and less than 1, corresponds to the ideal dehazing region, while the low-weight region, that is, the region with a weight value greater than 0 and less than 0.3, is the region with an unsatisfactory dehazing.

[0059] S3.4 The numerical processing part is divided into inverse processing and spatial regularization constraint part. Since the cross-attention matrix A is used to highlight the ideal dehazing region, in order to analyze the region with unsatisfactory dehazing, A is transformed by digital reciprocal as follows:

[0060]

[0061] Here, ε is a small constant used to prevent division by zero errors; the originally ideal dehazing region is compressed, and the originally unideal dehazing region is magnified. The resulting A′ can partially reflect the positional relationship between the unideal dehazing regions. To further enhance the model's understanding of the positional relationship between the unideal dehazing regions, spatial regularization is introduced as a constraint, that is, spatial similarity calculation is performed on A′ to obtain the relative positional relationship between the unideal dehazing regions, and the spatial autocorrelation matrix is ​​calculated as follows:

[0062] S(i,j)=∑ k A′ ij A′ jk ;

[0063] Here, S(i,j) reflects the similarity between position i and position j; if S(i,j) is greater than 0.7, it is considered that i and j are very similar in the feature space, which means that the undesmised dehazing regions i and j are adjacent or have similar dehazing problems; that is, S(i,j) serves as a matrix to measure the spatial relationship between undesmised dehazing regions.

[0064] Furthermore, the inverse interaction connection module described in S4 enhances the interaction between the dehazed image and the depth image by learning the spatial similarity between them. The inverse interaction mechanism uses the spatial similarity matrix S. D The similarity matrix is ​​calculated by comparing the dehazed and unhazed images through an attention mechanism. It is used to reflect the spatial relationship between different pixels. The depth estimation and dehazing tasks are optimized under the guidance of the similarity matrix. The specific process is as follows:

[0065] S4.1 The depth estimation network minimizes the difference between the depth estimate and the true depth map of the dehazed image, while optimizing the model through spatial structure similarity. The loss function is:

[0066]

[0067] Among them, S D It is the spatial similarity matrix calculated by inverse cross-attention between the dehazed and hazy images; Mu * This refers to the depth dehazed map obtained by processing the dehazed image through a depth estimation network in the downstream task; M u This refers to the depth map obtained by using a depth estimation network on a real, fog-free image. This refers to the depth map obtained by performing depth estimation on the initially input foggy map in the upstream task; α and β are balancing parameters; Mu * -Mu refers to the depth estimation error after dehazing, constraining the depth estimation map Mu of the dehazed image. * A depth map Mu that more closely resembles a real fog-free map. This is to obtain the depth estimation error before dehazing, so that the depth estimation map of the foggy image is obtained. A depth map M that is as close as possible to a real fog-free map u Sim(D) is a similarity matrix based on depth estimation, and its calculation process is as follows:

[0068] a. Extract features from the depth map D, i.e., the depth map of the dehazed image: in, Lightweight convolutional or low-pass filters are used to extract deep features;

[0069] b. Use cosine similarity to calculate the similarity between each pixel in the depth map:

[0070] Among them, Sim(D) i,j F represents the similarity between row index i and column index j. i and F j These are the depth feature vectors at indices i and j, used to measure the consistency of depth distribution across different regions;

[0071] c. To ensure the stability of the similarity matrix, Softmax normalization is performed:

[0072]

[0073] S4.2. The dehazing network is then optimized by addressing regions with poor depth estimation, allowing the network to actively learn about the spatial structure of these regions. The loss function for the depth network optimization is shown below:

[0074] L dehaz =||S D1 -Sim(D)1||1+α||u * -u||;

[0075] Among them, S D1 It is the spatial similarity matrix calculated by inverse cross-attention between the depth maps of the dehazed image and the depth maps of the hazy image; u * α is the dehazed image, u is the true haze-free image; α is the balance parameter.

[0076] A dehazing system for mine dust and fog images based on three-dimensional inverse interaction cross-linking includes an encoder-decoder architecture, in which a three-dimensional attention extraction module, an inverse cross-linking regularization attention module, and an inverse interaction connection module are provided;

[0077] The aforementioned 3D attention extraction module is used to jointly extract the spatial features and depth information of foggy images, i.e., the distribution and concentration of fog and haze, to achieve modeling and perception of the 3D structure of the image;

[0078] The inverse cross-regularization attention module is used to enhance the response of the dehazing error region through inverse numerical values, and combined with the spatial regularization strategy, strengthens the spatial perception and relative position modeling of the error region.

[0079] The inverse interaction connection module is used to calculate the error similarity matrix between the dehazed map and the depth map, guiding the two tasks to achieve dynamic bidirectional interaction during the training phase. At the same time, a loss function based on spatial regularization is designed to introduce error information into the feature learning phase.

[0080] This invention incorporates a 3D attention extraction module into the u-net network architecture, along with a spatially regularized inverse cross-attention module and an inverse interaction connection module in the downstream task. The 3D attention extraction module introduces spatial-depth modeling into the attention mechanism, effectively enhancing the model's ability to perceive non-uniform fog and providing clearer, more discriminative features for subsequent defogging. The spatially regularized inverse cross-attention module dynamically and adaptively highlights areas with residual fog, artifacts, and contrast loss, while strengthening the depth estimation network's response to these areas, thus improving the overall performance and robustness of the dual tasks. The inverse interaction connection module not only improves the collaborative efficiency of the defogging and depth estimation networks but also solves the problem of error information not being fed back to the feature layer, achieving deeper and more efficient dual-task optimization and improving the model's generalization ability in complex scenes. This invention achieves close collaborative optimization and performance improvement between image defogging and depth estimation tasks, enhancing the model's defogging effect and generalization performance, as well as the accuracy of monitoring camera systems and intelligent detection systems in mines, thereby improving the level of coal mine safety production. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the workflow of the method of the present invention;

[0082] Figure 2 This is a schematic diagram of the three-dimensional attention module and the dilated convolution module structure of the present invention;

[0083] Figure 3 This is a schematic diagram of the coordinate attention unit structure of the present invention;

[0084] Figure 4 This is a schematic diagram of the workflow of the spatial regularization inverse cross-attention module of the present invention;

[0085] Figure 5 This is a schematic diagram of the reverse interactive connection module of the present invention;

[0086] Figure 6(a) is an indoor image with fog, (b) is an image after defogging with FFA-net, (c) is an image after defogging with ConvIR-net, (d) is an image after defogging with GridDehazeNet, and (e) is an image after defogging using the present invention. Detailed Implementation

[0087] The invention will now be further described with reference to the accompanying drawings.

[0088] like Figure 1 As shown, a method for dehazing mine dust and fog images based on three-dimensional inverse interaction cross-section includes the following steps:

[0089] S1. Add a 3D attention extraction module to the u-net network architecture, and add a spatially regularized inverse cross-attention module and an inverse interactive connection module to the downstream task.

[0090] S2. In the upstream task, 3D attention is extracted from the fusion tensor of the foggy depth map obtained by the depth estimation network and the preliminary feature map obtained by the u-net network. The form of dilated 3D convolution + coordinate attention is used to further extract 3D information.

[0091] S3. In the downstream task, on the one hand, the dehazed image output by the u-net network and the real image are subjected to spatial regularization inverse cross attention, and the regions with unsatisfactory dehazing are highlighted through inverse numerical processing. At the same time, a spatial regularization module is added to highlight the relative positional relationship between the regions with unsatisfactory dehazing. The spatial regularization loss function in the depth estimation network guides the depth estimation network to focus on the regions with unsatisfactory residual fog, thereby strengthening the mutual learning between the networks.

[0092] On the other hand, the dehazed image is processed by a depth estimation network to obtain a dehazed depth map. This map is then subjected to spatial regularization inverse cross-attention with the real foggy depth map to highlight the regions with poor depth estimation and their relative positional relationships. The spatial regularization loss function of the dehazed network is used to guide the optimization of the network, making it more focused on learning the spatial structure of the regions with poor depth estimation and thus promoting the two-stage task.

[0093] S4. Input the obtained regions with global positional relationships and regions with poor depth estimation of the dehazed image into the inverse interactive connection module. Through the constraint of regularized similarity loss function, the regions with poor dehazed image and regions with poor depth estimation learn each other's spatial structure, and at the same time optimize the mutual network, namely the depth estimation network and the dehazed network.

[0094] Furthermore, the areas with unsatisfactory dehazing refer to areas with obvious fog residue, insufficient detail recovery, blurred edges, color distortion, or reduced local contrast, and areas with a peak signal-to-noise ratio (PSNR) below 25dB or a structural similarity index (SSIM) below 0.9. Conversely, areas with ideal dehazing are considered ideal areas. The areas with unsatisfactory depth in the dehazed image refer to areas where the depth value deviates from the true value, the depth boundary is blurred, the local depth continuity is disrupted, or there are abnormal jumps. Conversely, areas with ideal depth in the dehazed image are considered ideal areas.

[0095] The u-net network architecture in S1 includes three encoders and two decoders. Each encoder and decoder includes a ResBlock block. A three-dimensional attention extraction module is added after each ResBlock block. The first attention module is used to receive the depth information of the feature map. Before the first ResBlock block is the fusion tensor of the depth map and the feature map, as well as the multi-scale feature extraction module u-net. Through the extraction of multi-scale three-dimensional attention, the network extracts depth and spatial features.

[0096] like Figure 2 As shown, the 3D attention extraction module in S2 is an adaptive multi-scale 3D attention module, which includes a channel fusion layer, a convolutional layer, a normalization layer, a linear layer, a 3D dilated convolution, and coordinate attention units; the working process of the 3D attention extraction module is as follows:

[0097] S2.1 After merging via Concat channels, the output is denoted as F1:

[0098] F1 = Concat(Depth1 + u1 + u2);

[0099] Wherein, Concat is the channel fusion layer, Depth1 is the output image of the foggy image after passing through the depth estimation network, u1 is the feature map extracted by the multi-scale feature extraction module u-net network, u2 is the original color image; + indicates channel stacking;

[0100] S2.2 Initial feature extraction is performed using the Conv2d layer, with the following expression:

[0101] F2 = σ(W1*x + b1);

[0102] Where X is the input image, W1 and b1 are two-dimensional convolution kernel parameters, * represents the convolution operation, and σ is a non-linear activation function;

[0103] S2.3. Normalization layer norms accelerate model convergence and stability, as shown in the following expression:

[0104]

[0105] Where μ and δ are the mean and standard deviation calculated for the current batch, respectively, and γ and β are learnable parameters.

[0106] S2.4 The network learning extracts features at different levels through three parallel linear layer branches. The outputs of the different linear layers are A1, A2, and A3, as shown in the following expressions:

[0107] A1 = W2 * F' + b2;

[0108] A2 = W3 * F' + b3;

[0109] A3 = W4 * F' + b4;

[0110] Where W2, W3, W4 and b2, b3, b4 are the weights and biases of A1, A2, and A3, respectively;

[0111] S2.5. Further extract depth and spatial feature information through 3D convolutional layers. The outputs of different 3D dilated convolutional layers are C1, C2, and C3, respectively, and their expressions are as follows:

[0112] C1 = σ(W5*A1 + b5);

[0113] C2 = σ(W6*A2 + b6);

[0114] C3 = σ(W7*A3 + b7);

[0115] Where W5, W6, and W7 refer to convolutional kernels with different dilation rates; b5, b6, and b7 are the biases of convolutional layers with different dilation rates; and σ is a nonlinear activation function.

[0116] After channel fusion, feature mappings for different branches are obtained:

[0117] F out =Concat(C1+C2+C3);

[0118] S2.6, such as Figure 3 As shown, the coordinate attention unit includes horizontal and vertical global average pooling layers, a spatial fusion layer, a regular convolutional layer, a batch normalization layer, and a nonlinear layer; the expression for the working process of the horizontal and vertical global average pooling layers is as follows:

[0119]

[0120] Where w and H represent the width and height of the feature map, respectively; i and j represent the row and column indices, respectively;

[0121] S2.7. Perform matrix transpose on 1*W*C and then concatenate according to spatial order. The expression is:

[0122] f=σ(F2([Z h Z w ]));

[0123] Here, F2 refers to the spatial concatenation and fusion of the horizontal and vertical weight matrices, and Z... h Z w This refers to the transpose of the weight matrix obtained from horizontal and vertical global average pooling; σ refers to the non-linear activation function.

[0124] S2.8. To accelerate training, normalization and non-linear activation are used. The data is further divided into H*1*C and W*1*C, and sigmoid() is applied to generate two attention vectors, g and g, respectively. h and g w Among them, g h ∈C*1*W、g w ∈C*1*h, the expression is as follows:

[0125] g h =σ(F h (f h ));

[0126] g w =σ(F w (f w ));

[0127] F h (f h This refers to further dividing the fusion weight matrix f obtained above into H*1*C feature column vectors in the form of column vectors; F w (f w This refers to further dividing the fusion weight matrix f obtained above into W*1*C feature row vectors in the form of row vectors; σ is a non-linear activation function;

[0128] The output is:

[0129] Where i and j represent the row index and column index, respectively; x c (i,j) refers to the feature map of the original input. This refers to the attention weights of the column vectors obtained by applying coordinate attention to the original feature map. This refers to the row vector attention weights obtained by applying coordinate attention to the original feature map.

[0130] like Figure 4 As shown, the spatially regularized inverse cross-attention module in S3 includes a cross-attention unit, a numerical processing unit, and a spatial regularization unit, and its working process is as follows:

[0131] S3.1, D IG represents the image after dehazing. T This represents the corresponding real-world haze-free image. In a deep learning framework, the dehazed image D... I and true fog-free images G T After passing through the feature extraction network, the corresponding feature representations are obtained respectively:

[0132] F d =F(D I );

[0133] F c =F(G T );

[0134] Where F(.) represents the feature extraction network;

[0135] S3.2. Employ the Cross-Attention mechanism to calculate F. d and F c The attention weights between them are expressed as follows:

[0136] Q = F d W q ;

[0137] K = F c W k ;

[0138] V = F c W V ;

[0139] Where Q represents the query, K represents the key, V represents the value, and W represents the value. q W k W V All are linear transformation matrices. S3.3 and cross-attention are calculated as follows:

[0140]

[0141] Where A is the cross-attention matrix, with a size of H×W, representing the similarity between each region of the dehazed image and the real haze-free image; d k is the scaling factor of the key feature dimension, used to stabilize the gradient; since A reflects the degree of matching between the dehazed image and the haze-free image, its high-weight region, that is, the region with a weight value greater than 0.7 and less than 1, corresponds to the ideal dehazing region, while the low-weight region, that is, the region with a weight value greater than 0 and less than 0.3, is the region with an unsatisfactory dehazing.

[0142] S3.4 The numerical processing part is divided into inverse processing and spatial regularization constraint part. Since the cross-attention matrix A is used to highlight the ideal dehazing region, in order to analyze the region with unsatisfactory dehazing, A is transformed by digital reciprocal as follows:

[0143]

[0144] Here, ε is a small constant used to prevent division by zero errors; the originally ideal dehazing region is compressed, and the originally unideal dehazing region is magnified. The resulting A′ can partially reflect the positional relationship between the unideal dehazing regions. To further enhance the model's understanding of the positional relationship between the unideal dehazing regions, spatial regularization is introduced as a constraint, that is, spatial similarity calculation is performed on A′ to obtain the relative positional relationship between the unideal dehazing regions, and the spatial autocorrelation matrix is ​​calculated as follows:

[0145] S(i,j)=∑ k A′ ik A′ jk ;

[0146] Here, S(i,j) reflects the similarity between position i and position j; if S(i,j) is greater than 0.7, it is considered that i and j are very similar in the feature space, which means that the undesmised dehazing regions i and j are adjacent or have similar dehazing problems; that is, S(i,j) serves as a matrix to measure the spatial relationship between undesmised dehazing regions.

[0147] like Figure 5 As shown, the inverse interaction connection module in S4 enhances the interaction between the dehazed image and the depth image by learning the spatial similarity between them. The inverse interaction mechanism uses the spatial similarity matrix S. D The similarity matrix is ​​calculated by comparing the dehazed and unhazed images through an attention mechanism. It is used to reflect the spatial relationship between different pixels. The depth estimation and dehazing tasks are optimized under the guidance of the similarity matrix. The specific process is as follows:

[0148] S4.1 The depth estimation network minimizes the difference between the depth estimate and the true depth map of the dehazed image, while optimizing the model through spatial structure similarity. The loss function is:

[0149]

[0150] Among them, S D It is the spatial similarity matrix calculated by inverse cross-attention between the dehazed and hazy images; Mu * This refers to the depth dehazed map obtained by processing the dehazed image through a depth estimation network in the downstream task; M u This refers to the depth map obtained by using a depth estimation network on a real, fog-free image. This refers to the depth map obtained by performing depth estimation on the initially input foggy map in the upstream task; α and β are balancing parameters; Mu * -Mu refers to the depth estimation error after dehazing, constraining the depth estimation map Mu of the dehazed image.* A depth map Mu that more closely resembles a real fog-free map. This is to obtain the depth estimation error before dehazing, so that the depth estimation map of the foggy image is obtained. A depth map M that is as close as possible to a real fog-free map u Sim(D) is a similarity matrix based on depth estimation, and its calculation process is as follows:

[0151] a. Extract features from the depth map D, i.e., the depth map of the dehazed image: in, Lightweight convolutional or low-pass filters are used to extract deep features;

[0152] b. Use cosine similarity to calculate the similarity between each pixel in the depth map:

[0153] Among them, Sim(D) i,j F represents the similarity between row index i and column index j. i and F j These are the depth feature vectors at indices i and j, used to measure the consistency of depth distribution across different regions;

[0154] c. To ensure the stability of the similarity matrix, Softmax normalization is performed:

[0155]

[0156] S4.2. The dehazing network is then optimized by addressing regions with poor depth estimation, allowing the network to actively learn about the spatial structure of these regions. The loss function for the depth network optimization is shown below:

[0157] L dehaze =||S D1 -Sim(D)1||1+α||u * -u||;

[0158] Among them, S D1 It is the spatial similarity matrix calculated by inverse cross-attention between the depth maps of the dehazed image and the depth maps of the hazy image; u * α is the dehazed image, u is the true haze-free image; α is the balance parameter.

[0159] like Figure 6As shown in (a) to (e), to verify the dehazing effect of the present invention, the dehazing effect of indoor foggy images was compared on the SOTS-Indoor dataset. Among them, (a) is an indoor foggy image, (b) is an image after dehazing by FFA-net, (c) is an image after dehazing by ConvIR-net, (d) is an image after dehazing by GridDehazeNet, and (e) is an image after dehazing by the present invention. It can be seen that the dehazing effect of the present invention is significantly better than the effects shown in (b) to (d).

[0160] Table 1 compares the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the present invention with those of existing defogging methods. It is clear from the table that the defogging effect of the present invention is superior to that of the existing technology.

[0161] Table 1

[0162]

[0163]

Claims

1. A mine dust-mist image defogging method based on three-dimensional inverse interaction intersection, characterized in that, Comprise the following steps: S1, add a three-dimensional attention extraction module in the u-net network architecture, and add a spatial regularization inverse cross attention module and an inverse interaction connection module in the downstream task; S2, in the upstream task, perform three-dimensional attention extraction on the fusion tensor of the depth map with fog obtained through the depth estimation network and the preliminary feature map obtained through the u-net network, in the form of three-dimensional convolution with a hole and coordinate attention, to further extract three-dimensional information; S3, in the downstream task, on the one hand, the dehazed image output by the u-net network and the real image are subjected to spatial regularization inverse cross attention, and the ideal areas of dehazing are highlighted through reverse numerical processing, and a spatial regularization module is added to highlight the relative position relationship between the ideal areas of dehazing, the spatial regularization loss function in the depth estimation network is used to guide the depth estimation network to pay attention to the ideal areas of dehazing, and the mutual learning between the networks is strengthened; On the other hand, the dehazed depth map obtained through the depth estimation network is subjected to spatial regularization inverse cross attention with the real depth map with fog, the ideal areas of dehazing and the relative position relationship are highlighted, and the spatial regularization loss function of the dehazing network is used to constrain and guide the optimization of the dehazing network, so that the dehazing network pays more attention to the ideal areas of dehazing and the spatial structure learning of the ideal areas of dehazing, and the promotion of the two-stage task is realized; S4, the ideal areas of dehazing with global position relationship and the ideal areas of dehazing picture depth estimation are input into the inverse interaction connection module, and the regularization similarity loss function is used for constraint, to realize the mutual learning of the spatial structure of the ideal areas of dehazing and the ideal areas of depth estimation, and the optimization of the mutual network, i.e. the depth estimation network and the dehazing network.

2. The mine dust fog image defogging method based on three-dimensional inverse interaction intersection according to claim 1, characterized in that, The ideal areas of dehazing refer to areas with obvious residual fog, insufficient detail recovery, blurred edges, color distortion, or local contrast reduction, and the peak signal-to-noise ratio (PSNR) is less than 25 dB or the structural similarity index (SSIM) is less than 0.9, otherwise they are ideal dehazing areas; the ideal areas of dehazing picture depth refer to areas with depth values deviating from the true values, blurred depth boundaries, local depth continuity destruction, or abnormal jump phenomena, otherwise they are ideal dehazing picture depth areas. 3.The mine dust-mist image dehazing method based on three-dimensional inverse interaction intersection according to claim 1, characterized in that, The u-net network architecture in S1 comprises three encoders and two decoders, each encoder and decoder comprises a ResBlock block, a three-dimensional attention extraction module is added after each ResBlock block, the first attention module is used to receive the depth information of the feature map, and the first layer ResBlock block is followed by a depth map and feature map fusion tensor and a multi-scale feature extraction module u-net, so that the network extracts depth and spatial features through multi-scale three-dimensional attention extraction.

4. The mine dust fog image defogging method based on three-dimensional inverse interaction intersection according to claim 1, characterized in that, The three-dimensional attention extraction module in S2 is an adaptive multi-scale three-dimensional attention module, which comprises a channel fusion layer, a convolution layer, a normalization layer, a linear layer, a three-dimensional hole convolution, and a coordinate attention unit; the working process of the three-dimensional attention extraction module is as follows: S2.1, fusion through Concat channel, record output as : = ； wherein, is a channel fusion layer, is an output map of the foggy picture passing through the depth estimation network, is a feature map extracted by the multi-scale feature extraction module u-net network, is the original color image; + indicates channel stacking; S2.2, initial feature extraction is performed through the initial feature extraction Conv2d layer, and the expression is as follows: ； wherein x is an input image, and are two-dimensional convolution kernel parameters, * represents a convolution operation, and σ is a nonlinear activation function. S2.3, the convergence and stability of the model are accelerated through the normalization layer norm, and the expression is as follows: * ； where, and are the mean and standard deviation computed for the current batch, respectively, are learnable parameters, S2.4, Network learning extracts features of different levels through three parallel linear layer branches, and the outputs of different linear layers are respectively , , , the expression is as follows: = * + ； = * + ； = * F’+ ； wherein, , , and , are respectively , , weights and biases; S2.5, further extract the feature information of depth and space through a three-dimensional convolution layer, and the outputs of different three-dimensional hollow convolution layers are respectively , , the expression is as follows: = ) ； = ) ； = * + ) ； wherein, , , are biases of different dilated convolution layers; is a nonlinear activation function; After channel fusion, the feature maps of different branches are obtained: = Concat( ; S2.6, the coordinate attention unit includes horizontal and vertical global average pooling layers, a spatial fusion layer, a general convolution layer, a batch normalization layer and a nonlinear layer; wherein the expression of the working process of the horizontal and vertical global average pooling layers is as follows: ； ； wherein, and denote the width and height of the feature map, respectively; and denote the row and column indices, respectively. S2.7, to and respectively transposed and then spliced in the spatial dimension, the expression is: f= , ])) ； wherein, means spatially concatenating the horizontal and vertical weight matrices, , means transposing the weight matrix obtained by horizontal and vertical global average pooling; means a nonlinear activation function; S2.8, speed up the training by normalization and nonlinear activation, divide into H*1*C and W*1*C again plus sigmoid(), generate two attention vectors, respectively and , ∈C*1*W, ∈C*1*h, the expression is as follows: = ( ))； = ( ))； ) refers to the division of the fusion weight matrix f obtained above into characteristic column vectors of H*1*C again in the form of column vectors; ) refers to the division of the fusion weight matrix f obtained above into characteristic row vectors of W*1*C again in the form of row vectors; σ is a nonlinear activation function;​​ Output is: (i,j) = (i,j) (i,j) (i) (j) ; wherein, and denote row index and column index, respectively; (i,j) refers to the original input feature map, (i) refers to the column vector attention weight of the original feature map after coordinate attention, (j) refers to the row vector attention weight of the original feature map after coordinate attention.

5. The mine dust fog image defogging method based on three-dimensional inverse interaction intersection according to claim 4, characterized in that, The inverse cross attention module of the spatial regularization in S3 includes a cross attention unit, a numerical processing unit and a spatial regularization unit, and its working process is as follows: S3.1、 denotes the dehazed image, denotes the corresponding real haze-free image, under the deep learning framework, the dehazed image and the real haze-free image , after the feature extraction network, the corresponding feature representations are obtained respectively: = （ ）； = （ )； wherein, (.) represents a feature extraction network; S3.2, Cross-Attention mechanism is adopted to calculate and the attention weight between them, which is expressed as follows: Q= ； K= ； V= ； wherein Q denotes a query, K denotes a key, and V denotes a value, , , are linear transformation matrices, S3.3, the cross attention calculation is as follows: ； Wherein, A is a cross attention matrix, the size of which is HxW, representing the similarity degree of each area of the defogging image and the real haze-free image; is a scaling factor of the key feature dimension, used to stabilize the gradient; since A reflects the matching degree between the defogging image and the haze-free image, the high weight area, i.e. the area with a weight value greater than 0.7 and less than 1, corresponds to the ideal defogging area, while the low weight area, i.e. the area with a weight value greater than 0 and less than 0.3, is the area with non-ideal defogging. S3.4, the numerical processing part is divided into inverse processing and spatial regularization constraint part, since the cross attention matrix A is used to highlight the ideal area of fog removal, in order to analyze the non-ideal area of fog removal, the numerical inverse transformation of A is as follows: ； wherein, is a small constant to prevent division by zero error; the originally defogging ideal region is compressed, the originally defogging non-ideal region is enlarged, and the obtained can partially reflect the positional relationship of the mutually defogging non-ideal regions, in order to further strengthen the understanding of the positional relationship of the defogging non-ideal regions by the model, spatial regularization is introduced for constraint, that is, the spatial similarity of is calculated, the relative positional relationship between the defogging non-ideal regions is obtained, and the spatial autocorrelation matrix is calculated as follows: S(i,j)= ; Wherein, S(i,j) reflects the similarity between position i and position j; if S(i,j) is greater than 0.7, it is considered that i and j are very similar in the feature space, then it is considered that the non-ideal area i and j of fog removal are adjacent or have similar fog removal problems; that is, S(i,j) is used as a matrix for measuring the spatial relationship between non-ideal areas of fog removal.

6. The mine dust fog image defogging method based on three-dimensional inverse interaction intersection according to claim 5, characterized in that, The inverse interaction connection module in S4 enhances the interaction between the defogging image and the depth image by learning the spatial similarity therebetween, and the inverse interaction mechanism uses the spatial similarity matrix The spatial relationship between different pixels is calculated by the attention mechanism of the defogging image and the haze-free image, and the depth estimation and the defogging task are optimized under the guidance of the similarity matrix, and the specific process is as follows: S4.1, the depth estimation network minimizes the difference between the depth estimation of the fog removal image and the real depth map, and optimizes the model through spatial structure similarity, and the loss function is as follows: - M -Mu| ; wherein, is the spatial similarity matrix calculated by the inverse cross-attention of the defogged image and the haze-free image; M refers to the depth defogged image obtained by the depth estimation network on the defogged picture in the downstream task; refers to the depth map obtained by the depth estimation network on the real haze-free picture, refers to the depth map obtained by the depth estimation network on the haze image input at the beginning in the upstream task; and is a balance parameter; M -Mu refers to the depth estimation error after defogging, which constrains the depth estimation map M of the defogged image to be closer to the depth map Mu of the real haze-free image; is to obtain the depth estimation error before defogging, so that the depth estimation map M of the haze image is as close as possible to the depth map of the real haze-free image ; is a similarity matrix based on depth estimation, and the calculation process is: a. Feature extraction on the depth map D, i.e. the depth map of the dehazed picture: D) wherein, D) is a light-weight convolution or low-pass filter to extract depth features; b. Cosine similarity is used to calculate the similarity between each pixel of the depth map: Sim(D = ; where Sim(D denotes the similarity between row index i and column index j, and are the depth feature vectors at row index i and column index j, respectively, which measure the consistency of different regions in the depth distribution. c. In order to ensure the stability of the similarity matrix, Softmax normalization is performed: ； S4.2, the depth estimation non-ideal area reversely optimizes the fog removal network, so that the fog removal network actively learns the spatial structure of the depth estimation non-ideal area; the loss function of the depth network optimization is as follows: = - + - ； wherein, is a spatial similarity matrix computed from the depth map of the defogged image and the depth map of the defogged image inverse cross-attention; is a defogged image, is a real fog-free image; is a balancing parameter.

7. A mine dust fog image defogging system based on three-dimensional inverse cross-interaction cross, which is used for the mine dust fog image defogging method based on three-dimensional inverse cross-interaction cross as claimed in any one of claims 1 to 6, characterized in that, The encoder-decoder architecture is included, wherein the three-dimensional attention extraction module, the inverse cross regularization attention module and the inverse interaction connection module are arranged in the encoder-decoder architecture; The three-dimensional attention extraction module is used to jointly extract the spatial features and depth information of the foggy image, that is, the haze distribution concentration, so as to realize the modeling and perception of the three-dimensional structure of the image. The inverse cross regularization attention module is used to enhance the error area response through inverse numerical enhancement, and combines the spatial regularization strategy to strengthen the spatial perception and relative position modeling of the error area. The inverse interaction connection module is used to calculate the error similarity matrix between the fog removal image and the depth image, guide the two tasks to realize dynamic bidirectional interaction in the training stage, and design a loss function based on spatial regularization to introduce error information into the feature learning stage.

Images