Mine dust fog image defogging method and system based on three-dimensional inverse interactive intersection

Abstract

The invention discloses a mine dust fog image defogging method and system based on three-dimensional inverse interactive intersection, and the method comprises the steps: adding a three-dimensional attention extraction module in a u-net network architecture, and adding a space regularization inverse cross attention and inverse interactive connection module in a downstream task; the three-dimensional attention extraction module enhances the sensing ability of the model to non-uniform haze; the spatial regularization inverse cross attention module dynamically and adaptively highlights residual fog, artifacts and contrast loss areas, enhances the response of the depth estimation network to the areas, and improves the overall performance and robustness of double tasks; the inverse interactive connection module improves the cooperation efficiency of the defogging and depth estimation network, and also solves the problem that error information cannot be fed back to a feature layer. The system is an encoder-decoder architecture, and a three-dimensional attention extraction module, an inverse cross regularization attention module and an inverse interaction connection module are arranged in the encoder-decoder architecture. According to the invention, the accuracy of a monitoring camera system and an intelligent detection system in a mine is improved.

Description

Technical Field

[0001] The present invention relates to a mine dust and fog image defogging method and system based on three-dimensional inverse interactive crossover, belonging to the technical field of image defogging. Background Art

[0002] In the underground coal mine environment, coal ash and dust generated by long-term operations are diffused in the air, and the mine operation area widely uses a spray dust reduction system, which makes the air filled with high-concentration dust fog. In addition, there are complex lighting conditions inside the mine, including mining lamps, LED lighting, reflective material walls, etc. These factors cause the image quality captured by the imaging equipment to deteriorate seriously, which is manifested as follows: (1) Low contrast: Coal dust absorbs light, resulting in dark overall imaging and blurred target edges; (2) Local overexposure and halo phenomenon: Strong light sources such as mining lamps will form halo (Halation) phenomenon in foggy environments, increasing the degree of image blur; (3) Dense fog residue in local areas: The fog droplets in the spray dust reduction area are larger, and light scattering is enhanced, which significantly reduces the visibility in the local area. These problems directly affect the operation of the monitoring camera system and intelligent detection system in the mine, seriously interfering with the accuracy of tasks such as target detection, personnel identification, and equipment monitoring.

[0003] To address the above issues, depth estimation tasks in recent years have been based on enhanced dehazing modeling, but the structural information of the residual area has not been effectively utilized, and task collaboration is relatively limited. The Depth Information AssistedCollaborative Mutual Promotion Network joint training framework proposed at CVPR2024 has improved performance to a certain extent, but it still relies on static weighting and differential mechanisms, lacks three-dimensional structure modeling and feature level interaction, and limits the deep fusion between tasks. In summary, the current two-stage task model still has obvious shortcomings in terms of feature interaction, adaptability, and spatial structure modeling, making it difficult to meet the needs of closer collaborative optimization and performance improvement between image dehazing and depth estimation tasks. Summary of the Invention

[0004] The purpose of the present invention is to provide a mine dust and fog image defogging method and system based on three-dimensional inverse interactive crossover. This method and system can achieve close collaborative optimization and performance improvement between image defogging and depth estimation tasks, improve the accuracy of monitoring camera systems and intelligent detection systems in mines, and improve the safety production level of coal mines.

[0005] To achieve the above objectives, the present invention provides a mine dust and fog image defogging method based on three-dimensional inverse interaction, comprising the following steps:

[0006] S1. Add a 3D attention extraction module to the u-net network architecture, and add a spatially regularized inverse cross-attention module and an inverse cross-connection module to the downstream tasks;

[0007] S2. In the upstream task, 3D attention extraction is performed on the fused tensor of the foggy depth map obtained by the depth estimation network and the preliminary feature map obtained by the u-net network. The 3D information is further extracted using the form of dilated 3D convolution + coordinate attention.

[0008] In downstream tasks, we perform spatial regularization on the defogging image output by the u-net network and the real image by performing inverse cross-attention. We also perform inverse numerical processing to highlight the areas where defogging is not ideal. We also add a spatial regularization module to highlight the relative positional relationship between the areas where defogging is not ideal. We use the spatial regularization loss function in the depth estimation network to guide the depth estimation network to focus on the areas where residual fog is not ideal, thereby strengthening the mutual learning between the networks.

[0009] On the other hand, the dehazed image is passed through the depth estimation network to obtain a dehazed depth map. This map is then subjected to spatial regularization and inverse cross attention compared to the real foggy depth map. This method highlights the areas of poor depth estimation and their relative position relationships in the dehazed image. The spatial regularization loss function of the dehazed network is then used to constrain and guide the optimization of the dehazed network, allowing it to pay more attention to areas with poor depth estimation and learn the spatial structure of the poor areas, thus promoting the two-stage task.

[0010] S4. The obtained defogging imperfect areas with global positional relationships and the areas with imperfect depth estimation for the defogging images are input into the inverse interactive connection module. Through the constraints of the regularized similarity loss function, the defogging imperfect areas and the depth estimation imperfect areas can learn from each other's spatial structures, and at the same time, the mutual network, i.e., the depth estimation network and the defogging network, can be optimized.

[0011] Furthermore, the area with unsatisfactory defogging refers to an area with obvious residual fog, insufficient detail recovery, blurred edges, color distortion or local contrast reduction, and a peak signal-to-noise ratio PSNR lower than 25dB or a structural similarity index SSIM lower than 0.9. Otherwise, it is an ideal defogging area; the area with unsatisfactory depth of the defogging image refers to an area where the depth value deviates from the true value, the depth boundary is blurred, the local depth continuity is destroyed or there is an abnormal jump phenomenon. Otherwise, it is an area with ideal depth of the defogging image.

[0012] Furthermore, the u-net network architecture in S1 includes three encoders and two decoders, each encoder and decoder includes a ResBlock block, and a three-dimensional attention extraction module is added after each ResBlock block. The first attention module is used to receive the depth information of the feature map, and the first layer of ResBlock block is preceded by a fusion tensor of the depth map and the feature map and a multi-scale feature extraction module u-net. Through the extraction of multi-scale three-dimensional attention, the network extracts depth and spatial features.

[0013] Furthermore, the three-dimensional attention extraction module described in S2 is an adaptive multi-scale three-dimensional attention module, which includes a channel fusion layer, a convolution layer, a normalization layer, a linear layer, a three-dimensional hole convolution and a coordinate attention unit; the working process of the three-dimensional attention extraction module is as follows:

[0014] S2.1, after Concat channel fusion, the output is recorded as F1:

[0015] F1 = Concat(Depth1 + u1 + u2);

[0016] Concat is the channel fusion layer, Depth1 is the output image of the foggy image after the depth estimation network, u1 is the feature map extracted by the multi-scale feature extraction module u-net network, and u2 is the original color image; + indicates channel stacking;

[0017] S2.2, initial feature extraction is performed through the Conv2d layer, and the expression is as follows:

[0018] F2=σ(W1*x+b1);

[0019] Where X is the input image, W1 and b1 are the two-dimensional convolution kernel parameters, * represents the convolution operation, and σ is the nonlinear activation function;

[0020] S2.3, the normalization layer norm accelerates the convergence and stability of the model, and the expression is as follows:

[0021]

[0022] Among them, μ and δ are the mean and standard deviation calculated for the current batch, γ and β are learnable parameters.

[0023] S2.4. The network learns to extract features at different levels through three parallel linear layer branches. The outputs of different linear layers are A1, A2, and A3, respectively, and the expressions are as follows:

[0024] A1=W2*F'+b2;

[0025] A2=W3*F'+b3;

[0026] A3=W4*F'+b4;

[0027] Among them, W2, W3, W4 and b2, b3, b4 are the weights and biases of A1, A2, A3 respectively;

[0028] S2.5. Further extract the depth and spatial feature information through the three-dimensional convolution layer. The outputs of different three-dimensional hole convolution layers are C1, C2, and C3, respectively, and the expressions are as follows:

[0029] C1=σ(W5*A1+b5);

[0030] C2=σ(W6*A2+b6);

[0031] C3=σ(W7*A3+b7);

[0032] Among them, W5, W6, and W7 refer to convolution kernels with different dilation rates; b5, b6, and b7 are the biases of different dilated convolution layers; σ is a nonlinear activation function;

[0033] After channel fusion, the feature maps of different branches are obtained:

[0034] F out =Concat(C1+C2+C3);

[0035] S2.6. The coordinate attention unit includes horizontal and vertical global average pooling layers, spatial fusion layers, ordinary convolution layers, batch normalization layers, and nonlinear layers. The working process of the horizontal and vertical global average pooling layers is expressed as follows:

[0036]

[0037] Where w and H represent the width and height of the feature map respectively; i and j represent the row and column indices respectively;

[0038] S2.7. Transpose the matrix 1*W*C and then splice it in space. The expression is:

[0039] f=σ(F2([Z h ,Z w ]));

[0040] Among them, F2 refers to the spatial splicing and fusion of the horizontal and vertical weight matrices, Z h ,Z w Refers to the transposition operation of the weight matrix obtained by horizontal and vertical global average pooling; σ refers to the nonlinear activation function;

[0041] S2.8, through normalization and nonlinear activation to speed up training, again divided into H*1*C and W*1*C plus sigmoid(), generate two attention vectors, namely g h and g w , where g h ∈C*1*W,g w ∈C*1*h, the expression is as follows:

[0042] g h =σ(F h (f h ));

[0043] g w =σ(F w (f w ));

[0044] F h (f h ) means that the fusion weight matrix f obtained above is divided into H*1*C feature column vectors in the form of column vectors; F w (f w ) means dividing the fusion weight matrix f obtained above into W*1*C feature row vectors in the form of row vectors; σ is a nonlinear activation function;

[0045] The output is:

[0046] Among them, i and j represent the row index and column index respectively; x c (i,j) refers to the feature map of the original input, Refers to the column vector attention weight obtained by the coordinate attention of the original feature map, Refers to the row vector attention weight obtained by the coordinate attention of the original feature map;

[0047] Furthermore, the spatially regularized inverse cross-attention module in S3 includes a cross-attention unit, a numerical processing unit, and a spatial regularization unit, and its working process is as follows:

[0048] S3.1, D I Denotes the dehazed image, G T Represents the corresponding real fog-free image. Under the deep learning framework, the defogging image D I and the real fog-free image G T , after the feature extraction network, the corresponding feature representations are obtained:

[0049] F d =F(D I );

[0050] F c=F(G T );

[0051] Among them, F(.) represents the feature extraction network;

[0052] S3.2, using the cross-attention mechanism to calculate F d and F c The attention weight between is expressed as follows:

[0053] Q=F d W q ;

[0054] K=F c W k ;

[0055] V=F c W V ;

[0056] Among them, Q represents query, K represents key, V represents value, and W q 、W k 、W V Both are linear transformation matrices. S3.3, cross attention is calculated as follows:

[0057]

[0058] Among them, A is the cross attention matrix, whose size is H×W, which represents the similarity between each region of the defogging image and the real fog-free image; d k It is a scaling factor of the key feature dimension, which is used to stabilize the gradient. Since A reflects the degree of matching between the defogging image and the non-fogging image, its high-weight area, that is, the area with a weight value greater than 0.7 and less than 1, corresponds to the ideal defogging area, while the low-weight area, that is, the area with a weight value greater than 0 and less than 0.3, corresponds to the unsatisfactory defogging area.

[0059] S3.4. The numerical processing part is divided into the inverse processing and the spatial regularization constraint part. Since the cross attention matrix A is used to highlight the ideal defogging area, in order to analyze the unsatisfactory defogging area, the digital inverse transformation of A is performed as follows:

[0060]

[0061] Among them, ε is a small constant used to prevent division by zero errors; the originally ideal defogging area is compressed, and the originally poorly defogging area is magnified. The obtained A′ can partially reflect the positional relationship between the poorly defogging areas. To further strengthen the model's understanding of the positional relationship between the poorly defogging areas, spatial regularization is introduced to constrain it. That is, spatial similarity calculation is performed on A′ to obtain the relative positional relationship between the poorly defogging areas, and the spatial autocorrelation matrix is calculated as follows:

[0062] S(i,j)=∑ k A′ ij A′ jk ;

[0063] Among them, S(i,j) reflects the similarity between position i and position j. If S(i,j) is greater than 0.7, i and j are considered to be very similar in the feature space, which means that the defogging unsatisfactory areas i and j are adjacent or have similar defogging problems. That is, S(i,j) serves as a matrix to measure the spatial relationship between defogging unsatisfactory areas.

[0064] Furthermore, the inverse interaction connection module described in S4 enhances the interaction between the dehazed image and the depth image by learning the spatial similarity between the two. The inverse interaction mechanism uses the spatial similarity matrix S D , which is calculated from the defogging image and the fog-free image through the attention mechanism, is used to reflect the spatial relationship between different pixels. The depth estimation and defogging tasks are optimized under the guidance of the similarity matrix. The specific process is as follows:

[0065] S4.1. The depth estimation network minimizes the difference between the depth estimation of the dehazed image and the true depth map, while optimizing the model through spatial structural similarity. The loss function is:

[0066]

[0067] Among them, S D is the spatial similarity matrix obtained by the inverse cross attention calculation of the dehazed image and the haze-free image; Mu * Refers to the depth defogging image obtained by the depth estimation network in the downstream task; M u Refers to the depth map obtained by the depth estimation network of the real fog-free picture. Refers to the depth map obtained by depth estimation of the foggy image input at the beginning in the upstream task; α and β are balance parameters; Mu * -Mu refers to the depth estimation error after defogging, constraining the depth estimation map Mu of the defogging image * The depth map Mu is closer to the real fog-free image. The purpose is to obtain the depth estimation error before defogging, so that the depth estimation map of the foggy image The depth map M is as close as possible to the real fog-free image u ; Sim(D) is the similarity matrix based on depth estimation, and its calculation process is:

[0068] a. Perform feature extraction on the depth map D, i.e. the depth map of the dehazed image: in, It is a lightweight convolution or low-pass filter used to extract deep features;

[0069] b. Use cosine similarity to calculate the similarity between each pixel of the depth map:

[0070] Among them, Sim(D) i,j Represents the similarity between row index i and column index j, F i and F j is the depth feature vector at index i and j, which is used to measure the consistency of depth distribution in different regions;

[0071] c. In order to ensure the stability of the similarity matrix, Softmax normalization is performed:

[0072]

[0073] S4.2. The defogging network is optimized in the areas where depth estimation is not ideal, so that the defogging network actively learns the spatial structure of the areas where depth estimation is not ideal. The loss function of the depth network optimization is as follows:

[0074] L dehaz =||S D1 -Sim(D)1||1+α||u * -u||;

[0075] Among them, S D1 is the spatial similarity matrix obtained by inverse cross attention calculation of the depth map of the dehazed image and the depth map of the fog-free image; u * is the defogging image, u is the real fog-free image; α is the balance parameter.

[0076] A mine dust and fog image defogging system based on three-dimensional inverse crosstalk includes an encoder-decoder architecture in which a three-dimensional attention extraction module, an inverse crosstalk regularization attention module, and an inverse crosstalk connection module are provided;

[0077] The three-dimensional attention extraction module is used to jointly extract the spatial features and depth information of the foggy image, namely the haze distribution concentration, to achieve modeling and perception of the three-dimensional structure of the image;

[0078] The inverse cross-regularized attention module is used to enhance the dehazing error region response by inverse numerical value, and combined with the spatial regularization strategy, strengthen the spatial perception and relative position modeling of the error region;

[0079] The inverse interaction connection module is used to calculate the error similarity matrix between the dehazed image and the depth map, guiding the two tasks to achieve dynamic two-way interaction during the training phase. At the same time, a loss function based on spatial regularization is designed to introduce error information into the feature learning phase.

[0080] The present invention adds a three-dimensional attention extraction module to the u-net network architecture, and adds a spatially regularized inverse cross attention module and an inverse interactive connection module to the downstream tasks. The three-dimensional attention extraction module introduces spatial-depth modeling into the attention mechanism, effectively enhancing the model's perception of non-uniform haze and providing clearer and more discriminative features for subsequent defogging. The spatially regularized inverse cross attention module can dynamically and adaptively highlight areas such as residual fog, artifacts, and contrast loss, while strengthening the response of the depth estimation network to these areas, thereby improving the overall performance and robustness of the dual tasks. The inverse interactive connection module not only improves the collaborative efficiency of the defogging and depth estimation networks, but also solves the problem that error information cannot be fed back to the feature layer, achieving deeper and more efficient dual-task optimization and improving the generalization ability of the model in complex scenes. The present invention realizes close collaborative optimization and performance improvement between image defogging and depth estimation tasks, improves the defogging effect and generalization performance of the model, and the accuracy of the monitoring camera system and intelligent detection system in the mine, thereby improving the safety level of coal mine production. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a schematic diagram of the workflow of the method of the present invention;

[0082] Figure 2 Schematic diagram of the structure of the three-dimensional attention module hole convolution module of the present invention;

[0083] Figure 3 Schematic diagram of the coordinate attention unit structure of the present invention;

[0084] Figure 4 Schematic diagram of the workflow of the spatially regularized inverse cross-attention module of the present invention;

[0085] Figure 5 This is a schematic diagram of the reverse interactive connection module workflow of the present invention;

[0086] Figure 6(a) is an indoor foggy image, (b) is the image after FFA-net dehazing, (c) is the image after ConvIR-net dehazing, (d) is the image after GridDehazeNet dehazing, and (e) is the image after the dehazing of the present invention. DETAILED DESCRIPTION

[0087] The present invention will be further described below with reference to the accompanying drawings.

[0088] like Figure 1 As shown, a mine dust and fog image defogging method based on three-dimensional inverse interaction includes the following steps:

[0089] S1. Add a 3D attention extraction module to the u-net network architecture, and add a spatially regularized inverse cross-attention module and an inverse cross-connection module to the downstream tasks;

[0090] S2. In the upstream task, 3D attention extraction is performed on the fused tensor of the foggy depth map obtained by the depth estimation network and the preliminary feature map obtained by the u-net network. The 3D information is further extracted using the form of dilated 3D convolution + coordinate attention.

[0091] In downstream tasks, we perform spatial regularization on the defogging image output by the u-net network and the real image by performing inverse cross-attention. We also perform inverse numerical processing to highlight the areas where defogging is not ideal. We also add a spatial regularization module to highlight the relative positional relationship between the areas where defogging is not ideal. We use the spatial regularization loss function in the depth estimation network to guide the depth estimation network to focus on the areas where residual fog is not ideal, thereby strengthening the mutual learning between the networks.

[0092] On the other hand, the dehazed image is passed through the depth estimation network to obtain a dehazed depth map. This map is then subjected to spatial regularization and inverse cross attention compared to the real foggy depth map. This method highlights the areas of poor depth estimation and their relative position relationships in the dehazed image. The spatial regularization loss function of the dehazed network is then used to constrain and guide the optimization of the dehazed network, allowing it to pay more attention to areas with poor depth estimation and learn the spatial structure of the poor areas, thus promoting the two-stage task.

[0093] S4. The obtained defogging imperfect areas with global positional relationships and the areas with imperfect depth estimation for the defogging images are input into the inverse interactive connection module. Through the constraints of the regularized similarity loss function, the defogging imperfect areas and the depth estimation imperfect areas can learn from each other's spatial structures, and at the same time, the mutual network, i.e., the depth estimation network and the defogging network, can be optimized.

[0094] Furthermore, the area with unsatisfactory defogging refers to an area with obvious residual fog, insufficient detail recovery, blurred edges, color distortion or reduced local contrast, and an area with a peak signal-to-noise ratio PSNR lower than 25dB or a structural similarity index SSIM lower than 0.9. Otherwise, it is an ideal defogging area; the area with unsatisfactory depth of the defogged image refers to an area where the depth value deviates from the true value, the depth boundary is blurred, the local depth continuity is destroyed or there is an abnormal jump phenomenon. Otherwise, it is an area with ideal defogging depth.

[0095] The u-net network architecture in S1 includes three encoders and two decoders. Each encoder and decoder includes a ResBlock block. A three-dimensional attention extraction module is added after each ResBlock block. The first attention module is used to receive the depth information of the feature map, and the first layer of ResBlock blocks is preceded by a fusion tensor of the depth map and the feature map and a multi-scale feature extraction module u-net. Through the extraction of multi-scale three-dimensional attention, the network extracts depth and spatial features.

[0096] like Figure 2 As shown, the 3D attention extraction module described in S2 is an adaptive multi-scale 3D attention module, which includes a channel fusion layer, a convolution layer, a normalization layer, a linear layer, a 3D hole convolution and a coordinate attention unit; the working process of the 3D attention extraction module is as follows:

[0097] S2.1, after Concat channel fusion, the output is recorded as F1:

[0098] F1 = Concat(Depth1 + u1 + u2);

[0099] Concat is the channel fusion layer, Depth1 is the output image of the foggy image after the depth estimation network, u1 is the feature map extracted by the multi-scale feature extraction module u-net network, and u2 is the original color image; + indicates channel stacking;

[0100] S2.2, initial feature extraction is performed through the Conv2d layer, and the expression is as follows:

[0101] F2=σ(W1*x+b1);

[0102] Where X is the input image, W1 and b1 are the two-dimensional convolution kernel parameters, * represents the convolution operation, and σ is the nonlinear activation function;

[0103] S2.3, the normalization layer norm accelerates the convergence and stability of the model, and the expression is as follows:

[0104]

[0105] Among them, μ and δ are the mean and standard deviation calculated for the current batch, γ and β are learnable parameters.

[0106] S2.4. The network learns to extract features at different levels through three parallel linear layer branches. The outputs of different linear layers are A1, A2, and A3, respectively, and the expressions are as follows:

[0107] A1=W2*F'+b2;

[0108] A2=W3*F'+b3;

[0109] A3=W4*F'+b4;

[0110] Among them, W2, W3, W4 and b2, b3, b4 are the weights and biases of A1, A2, A3 respectively;

[0111] S2.5. Further extract the depth and spatial feature information through the three-dimensional convolution layer. The outputs of different three-dimensional hole convolution layers are C1, C2, and C3, respectively, and the expressions are as follows:

[0112] C1=σ(W5*A1+b5);

[0113] C2=σ(W6*A2+b6);

[0114] C3=σ(W7*A3+b7);

[0115] Among them, W5, W6, and W7 refer to convolution kernels with different dilation rates; b5, b6, and b7 are the biases of different dilated convolution layers; σ is a nonlinear activation function;

[0116] After channel fusion, the feature maps of different branches are obtained:

[0117] F out =Concat(C1+C2+C3);

[0118] S2.6, such as Figure 3 As shown in Figure 1, the coordinate attention unit includes horizontal and vertical global average pooling layers, spatial fusion layers, ordinary convolution layers, batch normalization layers, and nonlinear layers. The working process of the horizontal and vertical global average pooling layers is expressed as follows:

[0119]

[0120] Where w and H represent the width and height of the feature map respectively; i and j represent the row and column indices respectively;

[0121] S2.7. Transpose the matrix 1*W*C and then splice it in space. The expression is:

[0122] f=σ(F2([Z h ,Z w ]));

[0123] Among them, F2 refers to the spatial splicing and fusion of the horizontal and vertical weight matrices, Z h ,Z w Refers to the transposition operation of the weight matrix obtained by horizontal and vertical global average pooling; σ refers to the nonlinear activation function;

[0124] S2.8, through normalization and nonlinear activation to speed up training, again divided into H*1*C and W*1*C plus sigmoid(), generate two attention vectors, namely g h and g w , where g h ∈C*1*W,g w ∈C*1*h, the expression is as follows:

[0125] g h =σ(F h (f h ));

[0126] g w =σ(F w (f w ));

[0127] F h (f h ) means that the fusion weight matrix f obtained above is divided into H*1*C feature column vectors in the form of column vectors; F w (f w ) means dividing the fusion weight matrix f obtained above into W*1*C feature row vectors in the form of row vectors; σ is a nonlinear activation function;

[0128] The output is:

[0129] Among them, i and j represent the row index and column index respectively; x c (i,j) refers to the feature map of the original input, Refers to the column vector attention weight obtained by the coordinate attention of the original feature map, Refers to the row vector attention weight obtained by the coordinate attention of the original feature map.

[0130] like Figure 4 As shown, the spatially regularized inverse cross-attention module in S3 includes a cross-attention unit, a numerical processing unit, and a spatial regularization unit, and its working process is as follows:

[0131] S3.1, D IDenotes the dehazed image, G T Represents the corresponding real fog-free image. Under the deep learning framework, the defogging image D I and the real fog-free image G T , after the feature extraction network, the corresponding feature representations are obtained:

[0132] F d =F(D I );

[0133] F c =F(G T );

[0134] Among them, F(.) represents the feature extraction network;

[0135] S3.2, using the cross-attention mechanism to calculate F d and F c The attention weight between is expressed as follows:

[0136] Q=F d W q ;

[0137] K=F c W k ;

[0138] V=F c W V ;

[0139] Among them, Q represents query, K represents key, V represents value, and W q 、W k 、W V Both are linear transformation matrices. S3.3, cross attention is calculated as follows:

[0140]

[0141] Among them, A is the cross attention matrix, whose size is H×W, which represents the similarity between each region of the defogging image and the real fog-free image; d k It is a scaling factor of the key feature dimension, which is used to stabilize the gradient. Since A reflects the degree of matching between the defogging image and the non-fogging image, its high-weight area, that is, the area with a weight value greater than 0.7 and less than 1, corresponds to the ideal defogging area, while the low-weight area, that is, the area with a weight value greater than 0 and less than 0.3, corresponds to the unsatisfactory defogging area.

[0142] S3.4. The numerical processing part is divided into the inverse processing and the spatial regularization constraint part. Since the cross attention matrix A is used to highlight the ideal defogging area, in order to analyze the unsatisfactory defogging area, the digital inverse transformation of A is performed as follows:

[0143]

[0144] Among them, ε is a small constant used to prevent division by zero errors; the originally ideal defogging area is compressed, and the originally poorly defogging area is magnified. The obtained A′ can partially reflect the positional relationship between the poorly defogging areas. To further strengthen the model's understanding of the positional relationship between the poorly defogging areas, spatial regularization is introduced to constrain it. That is, spatial similarity calculation is performed on A′ to obtain the relative positional relationship between the poorly defogging areas, and the spatial autocorrelation matrix is calculated as follows:

[0145] S(i,j)=∑ k A′ ik A′ jk ;

[0146] Among them, S(i,j) reflects the similarity between position i and position j. If S(i,j) is greater than 0.7, i and j are considered to be very similar in the feature space, which means that the defogging unsatisfactory areas i and j are adjacent or have similar defogging problems. That is, S(i,j) serves as a matrix to measure the spatial relationship between defogging unsatisfactory areas.

[0147] like Figure 5 As shown, the inverse interaction connection module described in S4 enhances the interaction between the dehazed image and the depth image by learning the spatial similarity between the two. The inverse interaction mechanism uses the spatial similarity matrix S D , which is calculated from the defogging image and the fog-free image through the attention mechanism, is used to reflect the spatial relationship between different pixels. The depth estimation and defogging tasks are optimized under the guidance of the similarity matrix. The specific process is as follows:

[0148] S4.1. The depth estimation network minimizes the difference between the depth estimation of the dehazed image and the true depth map, while optimizing the model through spatial structural similarity. The loss function is:

[0149]

[0150] Among them, S D is the spatial similarity matrix obtained by the inverse cross attention calculation of the dehazed image and the haze-free image; Mu * Refers to the depth defogging image obtained by the depth estimation network in the downstream task; M u Refers to the depth map obtained by the depth estimation network of the real fog-free picture. Refers to the depth map obtained by depth estimation of the foggy image input at the beginning in the upstream task; α and β are balance parameters; Mu * -Mu refers to the depth estimation error after defogging, constraining the depth estimation map Mu of the defogging image* The depth map Mu is closer to the real fog-free image. The purpose is to obtain the depth estimation error before defogging, so that the depth estimation map of the foggy image The depth map M is as close as possible to the real fog-free image u ; Sim(D) is the similarity matrix based on depth estimation, and its calculation process is:

[0151] a. Perform feature extraction on the depth map D, i.e. the depth map of the dehazed image: in, It is a lightweight convolution or low-pass filter used to extract deep features;

[0152] b. Use cosine similarity to calculate the similarity between each pixel of the depth map:

[0153] Among them, Sim(D) i,j Represents the similarity between row index i and column index j, F i and F j is the depth feature vector at index i and j, which is used to measure the consistency of depth distribution in different regions;

[0154] c. In order to ensure the stability of the similarity matrix, Softmax normalization is performed:

[0155]

[0156] S4.2. The defogging network is optimized in the areas where depth estimation is not ideal, so that the defogging network actively learns the spatial structure of the areas where depth estimation is not ideal. The loss function of the depth network optimization is as follows:

[0157] L dehaze =||S D1 -Sim(D)1||1+α||u * -u||;

[0158] Among them, S D1 is the spatial similarity matrix obtained by inverse cross attention calculation of the depth map of the dehazed image and the depth map of the fog-free image; u * is the defogging image, u is the real fog-free image; α is the balance parameter.

[0159] like Figure 6As shown in (a) to (e), in order to verify the defogging effect of the present invention, the defogging effects of indoor foggy images are compared on the SOTS-Indoor dataset, where (a) is an indoor foggy image, (b) is the image after FFA-net defogging, (c) is the image after ConvIR-net defogging, (d) is the image after GridDehazeNet defogging, and (e) is the image after defogging of the present invention. It can be seen that the defogging effect of the present invention is significantly better than the effects shown in (b) to (d).

[0160] Table 1 shows a comparison between the present invention and the existing defogging method in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators. It can be clearly seen from the table that the defogging effect of the present invention is better than that achieved by the existing technology.

[0161] Table 1

[0162]

[0163]

Claims

1. A mine dust and fog image defogging method based on three-dimensional inverse interaction, characterized in that: The steps include: S1. Add a 3D attention extraction module to the u-net network architecture, and add a spatially regularized inverse cross-attention module and an inverse cross-connection module to the downstream tasks; S2. In the upstream task, 3D attention extraction is performed on the fused tensor of the foggy depth map obtained by the depth estimation network and the preliminary feature map obtained by the u-net network. The 3D information is further extracted using the form of dilated 3D convolution + coordinate attention. In downstream tasks, we perform spatial regularization on the defogging image output by the u-net network and the real image by performing inverse cross-attention. We also perform inverse numerical processing to highlight the areas where defogging is not ideal. We also add a spatial regularization module to highlight the relative positional relationship between the areas where defogging is not ideal. We use the spatial regularization loss function in the depth estimation network to guide the depth estimation network to focus on the areas where residual fog is not ideal, thereby strengthening the mutual learning between the networks. On the other hand, the dehazed image is passed through the depth estimation network to obtain a dehazed depth map. This map is then subjected to spatial regularization and inverse cross attention compared to the real foggy depth map. This method highlights the areas of poor depth estimation and their relative position relationships in the dehazed image. The spatial regularization loss function of the dehazed network is then used to constrain and guide the optimization of the dehazed network, allowing it to pay more attention to areas with poor depth estimation and learn the spatial structure of the poor areas, thus promoting the two-stage task. S4. The obtained defogging imperfect areas with global positional relationships and the areas with imperfect depth estimation for the defogging images are input into the inverse interactive connection module. Through the constraints of the regularized similarity loss function, the defogging imperfect areas and the depth estimation imperfect areas can learn from each other's spatial structures, and at the same time, the mutual network, i.e., the depth estimation network and the defogging network, can be optimized.

2. The mine dust and fog image defogging method based on three-dimensional inverse crosstalk according to claim 1 is characterized in that: The area where defogging is not ideal refers to an area with obvious residual fog, insufficient detail recovery, blurred edges, color distortion or local contrast reduction, and the peak signal-to-noise ratio PSNR is lower than 25dB or the structural similarity index SSIM is lower than 0.

9. Otherwise, it is an ideal defogging area; the area where the depth of the defogging image is not ideal refers to an area where the depth value deviates from the true value, the depth boundary is blurred, the local depth continuity is destroyed or there is an abnormal jump phenomenon. Otherwise, it is an area with ideal defogging depth.

3. The mine dust and fog image defogging method based on three-dimensional inverse crosstalk according to claim 1 is characterized in that: The u-net network architecture in S1 includes three encoders and two decoders. Each encoder and decoder includes a ResBlock block. A three-dimensional attention extraction module is added after each ResBlock block. The first attention module is used to receive the depth information of the feature map, and the first layer of ResBlock blocks is preceded by a fusion tensor of the depth map and the feature map and a multi-scale feature extraction module u-net. Through the extraction of multi-scale three-dimensional attention, the network extracts depth and spatial features.

4. The mine dust and fog image defogging method based on three-dimensional inverse crosstalk according to claim 1 is characterized in that: The 3D attention extraction module described in S2 is an adaptive multi-scale 3D attention module, which includes a channel fusion layer, a convolution layer, a normalization layer, a linear layer, a 3D hole convolution and a coordinate attention unit; the working process of the 3D attention extraction module is as follows: S2.1, after Concat channel fusion, the output is recorded as F1: F1 = Concat(Depth1 + u1 + u2); Concat is the channel fusion layer, Depth1 is the output image of the foggy image after the depth estimation network, u1 is the feature map extracted by the multi-scale feature extraction module u-net network, and u2 is the original color image; + indicates channel stacking; S2.2, initial feature extraction is performed through the Conv2d layer, and the expression is as follows: F2=σ(W1*x+b1); Where X is the input image, W1 and b1 are the two-dimensional convolution kernel parameters, * represents the convolution operation, and σ is the nonlinear activation function; S2.3, the normalization layer norm accelerates the convergence and stability of the model, and the expression is as follows: Among them, μ and δ are the mean and standard deviation calculated for the current batch, γ and β are learnable parameters. S2.

4. The network learns to extract features at different levels through three parallel linear layer branches. The outputs of different linear layers are A1, A2, and A3, respectively, and the expressions are as follows: A1=E2*F'+b2; A2=W3*F'+b3; A3=W4*F'+b4; Among them, W2, W3, W4 and b2, b3, b4 are the weights and biases of A1, A2, A3 respectively; S2.

5. Further extract the depth and spatial feature information through the three-dimensional convolution layer. The outputs of different three-dimensional hole convolution layers are C1, C2, and C3, respectively, and the expressions are as follows: C1=σ(W5*A1+b5); C2=σ(W6*A2+b6); C3=σ(W7*A2+b7); Among them, W5, W6, and W7 refer to convolution kernels with different dilation rates; b5, b6, and b7 are the biases of different dilated convolution layers; σ is a nonlinear activation function; After channel fusion, the feature maps of different branches are obtained: F out =Concat(C1+C2+C3); S2.

6. The coordinate attention unit includes horizontal and vertical global average pooling layers, spatial fusion layers, ordinary convolution layers, batch normalization layers, and nonlinear layers. The working process of the horizontal and vertical global average pooling layers is expressed as follows: Where w and H represent the width and height of the feature map respectively; i and j represent the row and column indices respectively; S2.

7. Transpose the matrix 1*W*C and then splice it in space. The expression is: f=σ(F2([Z h ,Z w ])); Among them, F2 refers to the spatial splicing and fusion of the horizontal and vertical weight matrices, Z h ,Z w Refers to the transposition operation of the weight matrix obtained by horizontal and vertical global average pooling; σ refers to the nonlinear activation function; S2.8, through normalization and nonlinear activation to speed up training, again divided into H*1*C and W*1*C plus sigmoid(), generate two attention vectors, namely g h and g w , where g h ∈C*1*W,g w ∈C*1*h, the expression is as follows: g h =σ(F h (f h )); g w =σ(F w (f w )); F h (f h ) means that the fusion weight matrix f obtained above is divided into H*1*C feature column vectors in the form of column vectors; F w (f w ) means dividing the fusion weight matrix f obtained above into W*1*C feature row vectors in the form of row vectors; σ is a nonlinear activation function; The output is: Among them, i and j represent the row index and column index respectively; x c (i,j) refers to the feature map of the original input, Refers to the column vector attention weight obtained by the coordinate attention of the original feature map, Refers to the row vector attention weight obtained by the coordinate attention of the original feature map.

5. The mine dust and fog image defogging method based on three-dimensional inverse crosstalk according to claim 1 is characterized in that: The spatially regularized inverse crisscross attention module in S3 includes a crisscross attention unit, a numerical processing unit, and a spatial regularization unit. Its working process is as follows: S3.1, D I Denotes the dehazed image, G T Represents the corresponding real fog-free image. Under the deep learning framework, the defogging image D I and the real fog-free image G T , after the feature extraction network, the corresponding feature representations are obtained: F d =F(D I ); F c =F(G T ); Among them, F(.) represents the feature extraction network; S3.2, using the cross-attention mechanism to calculate F d and F c The attention weight between is expressed as follows: Q=F d W q ; K=F c W k ; V=F c W V ; Among them, Q represents query, K represents key, V represents value, and W q 、W k 、W V are all linear transformation matrices, S3.3, cross attention is calculated as follows: Among them, A is the cross attention matrix, whose size is H×W, which represents the similarity between each region of the defogging image and the real fog-free image; d k It is a scaling factor of the key feature dimension, which is used to stabilize the gradient. Since A reflects the degree of matching between the defogging image and the non-fogging image, its high-weight area, that is, the area with a weight value greater than 0.7 and less than 1, corresponds to the ideal defogging area, while the low-weight area, that is, the area with a weight value greater than 0 and less than 0.3, corresponds to the unsatisfactory defogging area. S3.

4. The numerical processing part is divided into the inverse processing and the spatial regularization constraint part. Since the cross attention matrix A is used to highlight the ideal defogging area, in order to analyze the unsatisfactory defogging area, the digital inverse transformation of A is performed as follows: Among them, ε is a small constant used to prevent division by zero errors; the originally ideal defogging area is compressed, and the originally poorly defogging area is magnified. The obtained A′ can partially reflect the positional relationship between the poorly defogging areas. To further strengthen the model's understanding of the positional relationship between the poorly defogging areas, spatial regularization is introduced to constrain it. That is, spatial similarity calculation is performed on A′ to obtain the relative positional relationship between the poorly defogging areas, and the spatial autocorrelation matrix is calculated as follows: S(i,j)=∑ k A′ ik A j ′ k ; Among them, S(i,j) reflects the similarity between position i and position j. If S(i,j) is greater than 0.7, i and j are considered to be very similar in the feature space, which means that the defogging unsatisfactory areas i and j are adjacent or have similar defogging problems. That is, S(i,j) serves as a matrix to measure the spatial relationship between defogging unsatisfactory areas.

6. The mine dust and fog image defogging method based on three-dimensional inverse crosstalk according to claim 1 is characterized in that: The inverse interaction connection module described in S4 enhances the interaction between the dehazed image and the depth image by learning the spatial similarity between the two. The inverse interaction mechanism uses the spatial similarity matrix S D , which is calculated from the defogging image and the fog-free image through the attention mechanism, is used to reflect the spatial relationship between different pixels. The depth estimation and defogging tasks are optimized under the guidance of the similarity matrix. The specific process is as follows: S4.

1. The depth estimation network minimizes the difference between the depth estimation of the dehazed image and the true depth map, while optimizing the model through spatial structural similarity. The loss function is: Among them, S D is the spatial similarity matrix obtained by the inverse cross attention calculation of the dehazed image and the haze-free image; Mu * Refers to the depth defogging image obtained by the depth estimation network in the downstream task; M u Refers to the depth map obtained by the depth estimation network of the real fog-free picture. Refers to the depth map obtained by depth estimation of the foggy image input at the beginning in the upstream task; α and β are balance parameters; Mu * -Mu refers to the depth estimation error after defogging, constraining the depth estimation map Mu of the defogging image * The depth map Mu is closer to the real fog-free image; The purpose is to obtain the depth estimation error before defogging, so that the depth estimation map of the foggy image The depth map M is as close as possible to the real fog-free image u ; Sim(D) is the similarity matrix based on depth estimation, and its calculation process is: a. Perform feature extraction on the depth map D, i.e. the depth map of the defogging image: in, It is a lightweight convolution or low-pass filter used to extract deep features; b. Use cosine similarity to calculate the similarity between each pixel of the depth map: Among them, Sim(D) i,j Represents the similarity between row index i and column index j, F i and F j are the depth feature vectors at row index i and column index j, respectively, which are used to measure the consistency of depth distribution in different regions; c. In order to ensure the stability of the similarity matrix, Softmax normalization is performed: S4.

2. The defogging network is optimized in the areas where depth estimation is not ideal, so that the defogging network actively learns the spatial structure of the areas where depth estimation is not ideal. The loss function of the depth network optimization is as follows: L dehaze =||S D1 -Yes(D)1||1+α||u * -u||; Among them, S D1 is the spatial similarity matrix obtained by inverse cross attention calculation of the depth map of the dehazed image and the depth map of the fog-free image; u * is the defogging image, u is the real fog-free image; α is the balance parameter.

7. A mine dust and fog image defogging system based on three-dimensional inverse interaction, characterized in that: An encoder-decoder architecture is provided, wherein a three-dimensional attention extraction module, an inverse cross-regularization attention module, and an inverse interactive connection module are provided in the encoder-decoder architecture; The three-dimensional attention extraction module is used to jointly extract the spatial features and depth information of the foggy image, namely the haze distribution concentration, to achieve modeling and perception of the three-dimensional structure of the image; The inverse cross-regularized attention module is used to enhance the dehazing error region response by inverse numerical value, and combined with the spatial regularization strategy, strengthen the spatial perception and relative position modeling of the error region; The inverse interaction connection module is used to calculate the error similarity matrix between the dehazed image and the depth map, guiding the two tasks to achieve dynamic two-way interaction during the training phase. At the same time, a loss function based on spatial regularization is designed to introduce error information into the feature learning phase.

Images