Penetration perception Transform target detection method and device for dense fog degradation scene

Abstract

The invention provides a penetration perception Transform target detection method and device for a dense fog degradation scene. The method comprises the steps that a multi-scale visual compensation module is constructed, and the multi-scale visual compensation module is embedded into a Transform framework. And inputting the scene image into a target detection frame under the dense fog, and performing spatial feature extraction on the scene image. And carrying out feature redistribution on the first feature map by adopting a deformable attention mechanism, and carrying out nonlinear transformation on the second feature map so as to simulate a global illumination dispersion effect. And carrying out visibility mapping on the fourth feature map based on the fog layer penetration rate matrix, and carrying out residual connection on the target region features and the scene image to obtain a target in the dense fog scene. According to the method, the recovery capability of local details is enhanced while global modeling is carried out, the effect of visibility mapping is to enable FFN output to be more focused on a target area which still retains significant features after penetration, penetration perception can be fused from atmosphere degradation modeling, and the method has physical interpretation and multi-scale target perception capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a penetrating perception Transformer target detection method and device for thick fog degradation scenarios. BACKGROUND

[0002] In recent years, target detection technology has made significant progress in the field of computer vision, especially under the promotion of deep learning frameworks such as YOLO, FasterR-CNN, and DETR, efficient detection of traffic targets in clear images has been achieved. However, in actual intelligent traffic applications, especially in low-visibility extreme weather conditions such as morning fog, haze, sand, and rain and snow, traditional methods face the problem of performance degradation.

[0003] In a thick fog scenario, water vapor and particulate matter in the atmosphere can cause severe Mie scattering and Rayleigh scattering, resulting in low contrast, strong degradation, and structure loss in images. This complex degradation mechanism not only reduces image quality, but also directly damages the ability of deep learning models to distinguish key targets.

[0004] Existing target detection methods in thick fog scenarios include the following categories: (1) Image enhancement + detection two-stage method, combining classic image dehazing algorithms such as dark channel prior with target detection algorithms. Although this method improves image visual quality, the enhancement network and detection task are separated, which cannot perceive target semantic information, resulting in limited improvement in target detection performance. (2) Detection improvement network based on attention mechanism, some research attempts to introduce SE and CBAM channel attention structures in the detection model to enhance the salient target area in the degraded image. However, this method only optimizes from the feature selection perspective and cannot model the optical degradation process caused by thick fog, so it still performs unstable in extremely low visibility conditions. (3) Transformer visual modeling architecture, Vision Transformer methods such as Swin-Transformer and Deformable DETR have attracted attention due to their global perception ability. This method has certain advantages in identifying targets in thick fog scenarios. However, it still has problems in practical applications: the Transformer architecture is difficult to model the local spatial changes of fog, lacks a simulation penetration mechanism, and performs poorly in multi-scale small target recognition. (4) Methods that fuse modalities or deep priors, some methods attempt to use infrared images, depth maps, and other information to guide detection, but are limited by cost and environmental adaptability, and do not have the ability to be widely deployed in traffic scenarios.

[0005] In summary, there is an urgent need for a detection framework that models atmospheric degradation, can integrate penetrating perception, and has physical interpretability and multi-scale target perception ability. SUMMARY

[0006] To overcome the problems in the related art, the purpose of the present application is to provide a thick fog degradation scene-oriented penetration perception Transformer target detection method and device, wherein the method starts from atmospheric degradation modeling and can fuse penetration perception, has physical interpretability and multi-scale target perception capability.

[0007] A thick fog degradation scene-oriented penetration perception Transformer target detection method, comprising: constructing a multi-scale visual compensation module; embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene; obtaining a scene image under a thick fog scene, inputting the scene image into a thick fog target detection framework, performing spatial feature extraction on the scene image to obtain a first feature map; the thick fog target detection framework is composed of a plurality of penetration perception backbone networks; using a deformable attention mechanism to perform feature redistribution on the first feature map to obtain a second feature map; performing a nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map; calculating a fog layer penetration rate matrix according to a penetration rate function, the penetration rate function is used to simulate the visibility change of each pixel point of the scene image, performing visibility mapping on the fourth feature map based on the fog layer penetration rate matrix to obtain target region features; connecting the target region features and the scene image by residual connection to obtain a target under a thick fog scene.

[0008] In the preferred technical solution of the present application, the multi-scale visual compensation module is constructed, comprising: designing an s-Block module, an s-se-Block module and a Gate-Block module; wherein the s-Block module is used to improve the clarity of image local texture and object boundary; the s-Block module, the s-se-Block module and the Gate-Block module are combined to form a multi-scale visual compensation module.

[0009] In the preferred technical solution of the present application, the multi-scale visual compensation module is embedded into the Transformer architecture to obtain the penetration perception backbone network, comprising: connecting the input end of the first block encoding layer to the input layer and connecting the output end of the first encoding layer to the input end of the first s-Block module; connecting an output end of the first s-Block module to an input end of a second block encoding layer, connecting an output end of the second block encoding layer to an input end of a second s-Block module, and connecting an output end of the second s-Block module to an input end of a third block encoding layer; connecting an output end of the third block encoding layer to an input end of a third s-Block module, connecting an output end of the third s-Block module to an input end of the s-se-Block module, and connecting an output end of the s-se-Block module to an input end of a fourth block encoding layer; connecting an output end of the fourth block encoding layer to an input end of a fourth s-Block module, connecting an output end of the fourth s-Block module to an input end of the Gate-Block module, and connecting an output end of the Gate-Block module to an output layer.

[0010] In the preferred technical solution of the present application, the feature redistribution of the first feature map is performed by using a deformable attention mechanism to obtain a second feature map, which comprises: inputting the first feature map into a first subnetwork of a deformable attention module to generate a position offset of a sampling point; inputting the first feature map into a second subnetwork of the deformable attention module to generate an attention weight of the sampling point; adjusting the position of the sampling point by using the position offset of the sampling point to obtain a feature map after position adjustment; performing weighted summation on all sampling points of the feature map after position adjustment based on the attention weight of the sampling point to obtain the second feature map.

[0011] In the preferred technical solution of the present application, the position offset of the sampling point is generated by inputting the first feature map into the first subnetwork of the deformable attention module, which comprises: generating the position offset of the sampling point according to the following formula: ; wherein, is the position offset of the sampling point, Conv offset is the first subnetwork, the first subnetwork is used for predicting the position offset, and X1 is the first feature map.

[0012] In the preferred technical solution of the present application, the second feature map is subjected to a nonlinear transformation to simulate a global light diffusion effect to obtain a fourth feature map, which comprises: performing convolution on the second feature map by using a channel convolution kernel to perform channel transformation on the second feature map to obtain a third feature map; performing average pooling on the third feature map to obtain a pooled feature. The first feature map, the third feature map and the pooled feature are fused to obtain a fourth feature map.

[0013] In the preferred technical scheme of the present application, the calculation of the fog layer transmittance matrix according to the transmittance function comprises: The fog layer transmittance matrix is calculated according to the following formula: ; ; Wherein, d(x) is the perceived depth, the perceived depth is used to represent the blur degree of each position in the dense fog scene, is the fog intensity adjustment coefficient, max is the maximum value function, I c (x) is the pixel intensity of the cth channel of the pixel point of the image at position x, max c (I c (x)) is the maximum brightness channel value of the pixel point of the image at position x, c is the brightness channel serial number; T(x) is the fog layer transmittance matrix, exp is the exponential function, is the simulated fog concentration coefficient.

[0014] In the preferred technical scheme of the present application, the Gate-Block module comprises an r gate and an a gate, and the output of the Gate-Block module is: ; ; ; Wherein, G r (input) is the output of the r gate, is an activation function, input is the output of the fourth s-Block module, and FC is a full connection layer; is the output of the a gate, AvgPool is an average pooling operation, FFN is a forward feedback network, Output is the output of the Gate-Block module, and * is an element-wise multiplication.

[0015] In the preferred technical scheme of the present application, the visibility mapping of the fourth feature map based on the fog layer transmittance matrix obtains a target region feature, comprising: The target region feature is calculated according to the following formula: ; Wherein, Y is the target region feature, FFN is a forward feedback network, T(x) is a fog layer transmittance matrix, X4 is the fourth feature map, and. is a multiplication operation.

[0016] The application further provides a penetrating perception Transformer target detection device for a thick fog degradation scene, comprising: A visual compensation module construction unit is configured to construct a multi-scale visual compensation module. A module embedding unit is configured to embed the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is configured to enhance local features in a thick fog scene. A spatial feature extraction unit is configured to acquire a scene image in a thick fog scene, input the scene image into a target detection framework in a thick fog, perform spatial feature extraction on the scene image, and obtain a first feature map; the target detection framework in a thick fog is composed of a plurality of penetrating perception backbone networks. A feature redistribution unit is configured to perform feature redistribution on the first feature map using a deformable attention mechanism to obtain a second feature map. A nonlinear transformation unit is configured to perform nonlinear transformation on the second feature map to simulate a global light diffusion effect and obtain a fourth feature map. A visibility mapping unit is configured to calculate a fog layer transmittance matrix according to a transmittance function, the transmittance function being configured to simulate the fog layer visibility change of each pixel point of the scene image, and perform visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features. A residual connection unit is configured to perform residual connection on the target region features and the scene image to obtain a target in a thick fog scene.

[0017] The application has the following beneficial effects: The thick fog degradation scene-oriented penetration perception Transformer target detection method provided by the application comprises the following steps: constructing a multi-scale visual compensation module, embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene. Obtain a scene image under a thick fog scene, input the scene image into a target detection framework under a thick fog, perform spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under the thick fog is composed of a plurality of penetration perception backbone networks. On the basis of the multi-scale Transformer backbone, the ViT module is embedded in the feature layer of different resolutions, and the coupling attention mechanism is formed by combining the multi-scale visual compensation module, so that the recovery ability of local details is enhanced while the global modeling is performed. The deformable attention mechanism is used to redistribute the features of the first feature map to obtain a second feature map; the second feature map is subjected to nonlinear transformation to simulate the global light dispersion effect, and a fourth feature map is obtained. The nonlinear transformation comprises convolution, average pooling operation and weighted fusion, and the fourth feature map is a feature map with penetration perception capability. The fog layer penetration rate is calculated according to a penetration rate function, the penetration rate function is used to simulate the fog layer visibility change of each pixel point of the scene image, the fourth feature map is subjected to visibility mapping based on the fog layer penetration rate to obtain a target region feature. The fog layer penetration rate matrix is used to guide the feature weighting path in the FFN module, and the fog layer penetration rate matrix participates in the weight regulation in the forward propagation process, so that the output weight of the FFN structure is adjusted. The role of the visibility mapping is to make the FFN output more focused on the target region which still retains significant features after penetration. The target region feature and the scene image are connected by residual connection to obtain a target under a thick fog scene, and the residual connection plays a role in ensuring stable propagation of the feature. The method provided by the application is based on atmospheric degradation modeling, can integrate penetration perception, and has material explanation and multi-scale target perception capability. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the thick fog degradation scene-oriented penetration perception Transformer target detection method of the application; Figure 2 is a flowchart of the method for redistributing the features of the first feature map by using the deformable attention mechanism according to the application; Figure 3 is a structure diagram of the penetration perception backbone network according to the application; Figure 4 is a structure diagram of the s-Block module according to the application; Figure 5 is a structure diagram of the s-se-Block module according to the application; Figure 6 is a structure diagram of the Gate-Block module according to the application; Figure 7 is a first target recognition result graph in a thick fog scene of the present application; Figure 8 is a second target recognition result graph in a thick fog scene of the present application; Figure 9 is a third target recognition result graph in a thick fog scene of the present application; Figure 10 is a schematic diagram of a penetrating perception Transformer target detection device for a thick fog degradation scene of the present application. DETAILED DESCRIPTION

[0019] Preferred embodiments of the present application will be described in more detail with reference to the drawings. Although the preferred embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application is more thorough and complete, and the scope of the present application is fully conveyed to those skilled in the art.

[0020] Example 1 As shown in Figure 1 , the present embodiment provides a penetrating perception Transformer target detection method for a thick fog degradation scene, comprising: S1: constructing a multi-scale visual compensation module; S2: embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is used to enhance local features in a thick fog scene; S3: obtaining a scene image in a thick fog scene, inputting the scene image into a target detection framework under thick fog, performing spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under thick fog is composed of a plurality of penetrating perception backbone networks; S4: using a deformable attention mechanism to perform feature redistribution on the first feature map to obtain a second feature map; S5: performing a nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map; S6: calculating a fog layer penetration rate matrix according to a penetration rate function, the penetration rate function being used to simulate the change in visibility of each pixel point of the scene image, and performing visibility mapping on the fourth feature map based on the fog layer penetration rate matrix to obtain target region features; S7: connecting the target region features and the scene image in residual connection to obtain a target in a thick fog scene.

[0021] As shown in Figure 3As shown in the figure, the scene image is input into the first penetration perception backbone network of the target detection framework in dense fog, namely the first s-Block module of PenetrateViT, and 3×3 depth-separable convolution and 1×1 depth-separable convolution are performed on the scene image respectively. The results of the two depth-separable convolutions are then spliced ​​together and the spliced ​​result is input into the batch normalization neural network to extract the first feature map.

[0022] The extracting spatial features of the scene image to obtain a first feature map includes: A first feature map is extracted from the scene image using depthwise separable convolution according to the following formula: ; Among them, X1 is the first feature map, BN is the batch normalization neural network, DWConv 3×3 DWConv is a depth-wise separable convolution with a size of 3×3. 1×1 is a depthwise separable convolution with a size of 1×1, where X is the scene image. This embodiment extracts spatial features from the scene image, thereby achieving lightweight spatial perception modeling while preserving the original texture response.

[0023] The multi-scale visual compensation module is constructed, including: S11: Designing an s-Block module, an s-se-Block module, and a Gate-Block module; wherein the s-Block module is used to enhance the clarity of local textures and object boundaries in an image; S12: The s-Block module, the s-se-Block module, and the Gate-Block module are combined into a multi-scale visual compensation module.

[0024] The s-Block module simulates non-uniform spatial deformation through a deformable attention mechanism, adopts a penetration function to simulate optical degradation guidance, and strengthens the Transformer network's structural penetration perception capability in dense fog through a mechanism of joint modeling of local and global features, improving the model's detection performance for long-range, small-scale, and edge-blurred targets.

[0025] The deformable attention mechanism allocates attention only to key pixels, significantly reducing computational effort and providing greater robustness to deformation and scale changes. The deformable attention module uses 4-8 sampling points, with offsets varying with the input data. The receptive field is deformable, allowing for sampling features at different scales.

[0026] like Figure 2 As shown, the deformable attention mechanism is used to redistribute the features of the first feature map to obtain a second feature map, including: S41: input the first feature map into a first subnetwork of the deformable attention module to generate a position offset of the sampling point; S42: input the first feature map into a second subnetwork of the deformable attention module to generate an attention weight of the sampling point; S43: adjust the position of the sampling point by using the position offset of the sampling point to obtain a position-adjusted feature map; S44: based on the attention weight of the sampling point, perform weighted summation on all sampling points of the position-adjusted feature map to obtain a second feature map.

[0027] The calculation process of the deformable attention module is as follows: ; ; wherein, is the position offset of the sampling point, Conv offset is the first subnetwork, the first subnetwork is used for predicting the position offset, X1 is the first feature map; W is the attention weight of the sampling point, Conv weight is the second subnetwork. The first subnetwork Conv offset is used for predicting the position offset of each sampling point, and the output dimension is Kx2, which represents the displacement of the K sampling points in the x-axis direction and the y-axis direction. The second subnetwork Conv weight is used for predicting the attention weight w k of each offset sampling point, and the output dimension is K, which represents the attention weight of the K sampling points. The parameters of the first subnetwork and the second subnetwork are automatically learned from data by a back propagation algorithm, i.e. the parameters of the two subnetworks are trainable.

[0028] Preferably, the attention weight of each offset sampling point is normalized by using a normalized exponential function, so as to ensure that the sum of the attention weights of the K offset sampling points is 1. The normalized attention weight of the kth offset sampling point is calculated according to the following formula: ; wherein, is the normalized attention weight of the kth offset sampling point, exp is an exponential function, w k is the attention weight of the kth offset sampling point, w j is the attention weight of the jth offset sampling point, K is the total number of offset sampling points, and j is the serial number of the offset sampling point. The deformable attention mechanism can improve the modeling ability of boundary distortion and target offset in heavy fog.

[0029] After the K sampling points are adjusted, K offset sampling points are obtained. Based on the position offset of the sampling points and the attention weight of the sampling points, the second feature map is calculated using the following formula: ; ; Among them, X2 is the second feature map, DeformAttn is the output of the deformable attention module, X1 is the first feature map, is the position offset of the sampling point, is a matrix; w k is the attention weight of the k-th offset sampling point, p k is the coordinate of the kth sampling point in the first feature map, is the position offset of the kth sampling point in the first feature map, and is the multiplication operation. To adjust p k , to update the coordinates of the sampling points in the first feature map, obtain the coordinates of the offset sampling points in the first feature map, and the values ​​corresponding to the coordinates of all offset sampling points constitute the position-adjusted feature map. Optionally, replace Replace with .

[0030] The deformable attention mechanism can simulate the feature drift caused by optical scattering and refraction in dense fog. Compared with the ordinary attention mechanism, the deformable attention mechanism is more suitable for modeling the non-rigid changes in target morphology in dense fog, improving the spatial adaptability of the model in unstructured scenes.

[0031] To enhance physical interpretability, a penetration function is designed to simulate the visibility changes of each pixel in the image through the fog layer. The fog layer penetration matrix is ​​calculated based on the penetration function and is used to control weights during the forward propagation process. The visibility map allows the FFN output to focus more on the target area that retains significant features after penetration. The target area feature Y is residually connected to the scene image X, i.e., Z = X + Y, where Z is the target in the dense fog scene, to ensure stable feature propagation.

[0032] The thick fog degradation scene-oriented penetration perception Transformer target detection method provided by the embodiment comprises: constructing a multi-scale visual compensation module, embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene. A scene image under a thick fog scene is acquired, the scene image is input into a target detection framework under a thick fog, spatial feature extraction is performed on the scene image to obtain a first feature map; the target detection framework under the thick fog is composed of a plurality of penetration perception backbone networks. On the basis of a multi-scale Transformer backbone, a ViT module is embedded in a feature layer of different resolutions, and a coupling attention mechanism is formed in combination with a multi-scale visual compensation module to enhance the recovery capability of local details while globally modeling. A deformable attention mechanism is used to perform feature redistribution on the first feature map to obtain a second feature map; a nonlinear transformation is performed on the second feature map to simulate a global light dispersion effect, thereby obtaining a fourth feature map. The nonlinear transformation comprises convolution, average pooling operation and weighted fusion, and the fourth feature map is a feature map with penetration perception capability. A fog layer penetration rate is calculated according to a penetration rate function, the penetration rate function is used to simulate the fog layer visibility change of each pixel point of the scene image, and a visibility mapping is performed on the fourth feature map based on the fog layer penetration rate to obtain a target region feature. The fog layer penetration rate matrix is used to guide a feature weighting path in the FFN module, and the fog layer penetration rate matrix participates in weight regulation in the forward propagation process, thereby adjusting the output weight of the FFN structure. The role of the visibility mapping is to make the FFN output more focused on the target region that still retains significant features after penetration. The target region feature is connected with the scene image through residual connection to obtain a target under a thick fog scene, and the residual connection plays a role in ensuring stable feature propagation. The method provided by the present application can fuse penetration perception starting from atmospheric degradation modeling, and has material interpretability and multi-scale target perception capability.

[0033] Embodiment 2 The embodiment provides a thick fog degradation scene-oriented penetration perception Transformer target detection method, and the embodiment describes the differences from the embodiment 1 based on the embodiment 1. The method comprises: S1: constructing a multi-scale visual compensation module; S2: embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene. S3: acquiring a scene image under a thick fog scene, inputting the scene image into a target detection framework under a thick fog, performing spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under the thick fog is composed of a plurality of penetration perception backbone networks. S4: performing feature redistribution on the first feature map by using a deformable attention mechanism to obtain a second feature map; S5: performing nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map; S6: calculating a fog layer transmittance matrix according to a transmittance function used to simulate changes in visibility of the fog layer for each pixel point of the scene image, and performing visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features; S7: performing residual connection on the target region features and the scene image to obtain a target in a heavy fog scene.

[0034] The multi-scale visual compensation module comprises: S11: designing an s-Block module, an s-se-Block module, and a Gate-Block module; wherein the s-Block module is used to improve the definition of local textures and object boundaries of an image; S12: combining the s-Block module, the s-se-Block module, and the Gate-Block module to form a multi-scale visual compensation module.

[0035] Figure 4 The s-Block module is a core visual compensation structure in the application, and is designed to alleviate problems such as local texture blurring and structure boundary disappearance in a heavy fog environment. The s-Block module is embedded in the main body of the Transformer in a residual manner, and by introducing a deformable attention mechanism, an adaptive convolution combination, and a shallow feature fusion mechanism, local non-uniform light scattering characteristics in a heavy fog environment can be simulated, and physical enhancement perception of target edge regions can be realized. The processing of the s-Block module on data comprises the following steps: (1) performing spatial feature extraction on a scene image in a heavy fog environment to obtain a first feature map; (2) inputting the first feature map into a deformable attention module to obtain a second feature map; and (3) performing nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map.

[0036] The nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map comprises: S51: performing convolution on the second feature map by using a channel convolution kernel to perform channel transformation on the second feature map to obtain a third feature map; S52: performing average pooling on the third feature map to obtain a pooled feature; S53: fusing the first feature map, the third feature map, and the pooled feature to obtain a fourth feature map.

[0037] The fourth feature map is calculated according to the following formula: ; ; ; wherein X3 is the third feature map, Conv 1×1 is a convolution operation with a size of 1x1, X2 is the second feature map; X avg is the pooled feature, AvgPool is average pooling, X4 is the fourth feature map, is a fusion coefficient.

[0038] The second feature map is subjected to channel transformation by using a 1x1 convolution kernel. The 1x1 convolution kernel is used to change the channel number of the feature map under the premise that the image width W and height H remain unchanged. The global average pooling can simulate the global light diffusion effect in the atmospheric degradation image. The global light diffusion effect is a macroscopic phenomenon of light re-emission after multiple scattering in the medium. The first feature map, the third feature map and the pooled feature are fused by using weighted summation, and the fusion coefficient is adjusted to make the fourth feature map have a component of low-frequency diffuse reflection of global light, thereby enhancing the modeling ability of the model to the background bias.

[0039] The transmittance matrix of the fog layer is calculated according to the transmittance function, comprising: The transmittance matrix of the fog layer is calculated according to the following formula: ; ; wherein d(x) is a perceived depth, the perceived depth is used to represent the blur degree of each position in the dense fog scene, is a fog intensity adjustment coefficient, max is a maximum value function, I c (x) is the pixel intensity of the cth channel of the pixel point of the image at position x, max c (I c (x)) is the maximum brightness channel value of the pixel point of the image at position x. c is the brightness channel serial number; T(x) is the transmittance matrix of the fog layer, exp is an exponential function, is a simulated fog concentration coefficient,. is a multiplication operation.

[0040] The maximum value of one channel of the pixel point at position x, for example, the pixel point at the rowth row and the colth column in the RGB three channels of the scene image is calculated, wherein row≤H, col≤W, H is the height of the scene image, and W is the width of the scene image. The perceived depth d(x) represents the blur degree of the scene image at position x.

[0041] To enhance the physical interpretability, the transmittance function is designed to simulate the visibility change of each pixel in the image in the fog layer. The fog intensity adjustment coefficient is negatively correlated with the concentration of haze in the environment, that is, the higher the concentration of haze in the environment, the larger the fog intensity adjustment coefficient . The fog layer transmittance matrix T(x) is used to guide the feature weighting path within the FFN module, and the fog layer transmittance matrix corresponds to a transmittance map which participates in weight regulation in the process of forward propagation, thereby adjusting the output weight of the FFN structure. The FFN structure is a forward feedback network, which has the characteristic of one-way flow of data, that is, data is input from the input layer, passes through the hidden layer, and is output from the output layer, and the neurons in the same layer are fully connected.

[0042] The visibility mapping is performed on the fourth feature map based on the fog layer transmittance matrix to obtain target region features, including: The target region features are calculated according to the following formula: ; Where Y is the target region feature, FFN is a forward feedback network, T(x) is a fog layer transmittance matrix, X4 is the fourth feature map, and. is a multiplication operation.

[0043] The visibility mapping based on physical degradation makes the output of the FFN more focused on the target region that still retains significant features after penetration, and connects the target region features with the scene image in residual connection, thereby ensuring stable feature propagation. The following formula is used to calculate the target in a thick fog scene: Z = X + Y, where X is the scene image, Y is the target region feature, and Z is the target in the thick fog scene.

[0044] The s-Block of the present embodiment simulates non-uniform spatial deformation through a deformable attention mechanism, simulates optical degradation guidance through a transmittance function, and under the mechanism of local-global joint modeling, strengthens the structural penetration perception ability of the Transformer network in thick fog, and improves the detection performance of the model on distant, small-scale and edge blurred targets.

[0045] Embodiment 3 The present embodiment provides a penetration perception Transformer target detection method for thick fog degradation scenes, and describes the differences between the present embodiment and Embodiments 1 and 2 based on Embodiment 1. The method comprises: S1: constructing a multi-scale visual compensation module; S2: embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene; S3: acquire a scene image in a thick fog scene, input the scene image into a target detection framework under thick fog, perform spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under thick fog is composed of a plurality of the penetration perception backbone networks; S4: perform feature redistribution on the first feature map by using a deformable attention mechanism to obtain a second feature map; S5: perform nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map; S6: calculate a fog layer penetration rate matrix according to a penetration rate function, the penetration rate function is used to simulate the fog layer visibility change of each pixel point of the scene image, perform visibility mapping on the fourth feature map based on the fog layer penetration rate matrix to obtain a target region feature; S7: perform residual connection on the target region feature and the scene image to obtain a target in a thick fog scene.

[0046] The multi-scale visual compensation module is embedded into a Transformer architecture to obtain a penetration perception backbone network, including: S21: connect the input end of the first block encoding layer to the input layer, and connect the output end of the first encoding layer to the input end of the first s-Block module; S22: connect the output end of the first s-Block module to the input end of the second block encoding layer, and connect the output end of the second block encoding layer to the input end of the second s-Block module; S23: connect the output end of the second s-Block module to the input end of the third block encoding layer, connect the output end of the third block encoding layer to the input end of the third s-Block module, and connect the output end of the third s-Block module to the input end of the s-se-Block module; S24: connect the output end of the s-se-Block module to the input end of the fourth block encoding layer, connect the output end of the fourth block encoding layer to the input end of the fourth s-Block module, connect the output end of the fourth s-Block module to the input end of the Gate-Block module, and connect the output end of the Gate-Block module to the output layer.

[0047] The structure of the penetration perception backbone network is as shown in Figure 3 The target detection framework under thick fog includes a backbone network and a neck network, the backbone network is composed of a plurality of penetration perception backbone networks, i.e., PenetrateViT, the Transformer backbone of PenetrateViT is unfolded in a hierarchical structure, the input scene image is divided into N=HW / P 2A patch token is an image block tag.

[0048] Figure 5 The s-se-Block module is based on the s-Block module and introduces the SE attention mechanism to enhance dynamic response capabilities in the channel dimension. The SE attention mechanism can simulate the inter-channel saliency changes caused by differences in fog particle concentration distribution in dense fog environments, thereby performing weighted enhancement on the feature dimension of the target area.

[0049] Compared to the s-Block module, the core improvement of the s-se-Block module is that the input features are fed into the SE module before being fed into the main convolution path, and then participate in the final residual fusion after the channel-aware weights are extracted. The overall residual output is expressed as: ; Among them, Temp s-se-Block is the output of the s-se-Block module, FFN is the forward feedback network, When the input is the output of the third s-Block, the fog layer penetration matrix calculated by the penetration function is: is the output of the third s-Block, DeformAttn is deformable attention, BN is batch normalization network, f represents the mapping function, the mapping function f is the concatenation result of 3×3 depth-separable convolution and 1×1 depth-separable convolution, SE represents the squeeze excitation mechanism, Figure 5 The Y' in .

[0050] After introducing the SE mechanism, s-se-Block can not only cope with spatial blur degradation, but also dynamically model the physical phenomenon of channel importance shift, which is the non-uniformity of the fog layer. Channel importance shift refers to the situation where the brightness of the blue channel of the scene image in a dense scene is significantly lower than that of the red and green channels.

[0051] The Gate-Block module is used to simulate the dynamic penetration adjustment mechanism of image feature information in a dense fog environment. Its core idea is to perform gated selection of feature information based on the difficulty of fog penetration in different channels and regions to retain and enhance some feature information. The structure of the Gate-Block module is as follows: Figure 6 As shown in Figure 3, the Gate-Block module consists of two substructures: the residual gate unit r-Gate and the enhanced gate unit a-Gate. The residual gate unit r-Gate is used for information filtering, and the enhanced gate unit a-Gate is used for energy compensation, thus constructing a feature controller with active selective penetration characteristics.

[0052] The process of processing the input feature input of the Gate-Block module to obtain the output feature after the input feature input is processed by the Gate-Block module is as follows: The input feature input of the Gate-Block module is first processed by the r-Gate structure: ; wherein FC is a full connection layer, is an activation function such as a Sigmoid function, is the output of the residual gate unit r-Gate structure, that is, the output of the r gate, and input is the output of the fourth s-Block module.

[0053] The residual gate unit r-Gate structure is embedded with two nonlinear layers. The first nonlinear layer obtains an initial penetration control amount through FC, and the second nonlinear layer obtains a feature map after fusion of the input. The residual gate unit r-Gate structure plays a role in simulating information suppression of an image when the image is disturbed under dense fog conditions.

[0054] The enhanced gate unit a-Gate takes the output of the residual gate unit r-Gate as input. The enhanced gate unit a-Gate plays a role in guiding the characteristics to the global fog response area for compensation.

[0055] ; wherein, is the output of the enhanced gate unit a-Gate, that is, the output of the a gate, and AvgPool is an average pooling operation, is an activation function, and FFN is a forward feedback network, is the output of the residual gate unit r-Gate structure, that is, the output of the r gate.

[0056] The final fusion output of the Gate-Block module is: ; wherein Output is the output of the Gate-Block module, and * is an element-wise multiplication.

[0057] The function of the residual gate unit r-Gate is to model local penetration shielding. G r (input) is used to simulate the penetration suppression degree of image channel features in different areas of dense fog. The output of the residual gate unit r-Gate is a local penetration response map, which is explained as the difference in penetration ability of light on various channels such as color channels and texture channels. The function of the enhanced gate unit a-Gate is to model global fog field adjustment. G a (G rThe residual information and the feedforward enhancement path are fused, and the global average pooling is used to simulate the suppressive modulation of the fog field on the overall image brightness and texture distribution, so as to realize the modeling compensation of the diffuse light disturbance of the fog field.

[0058] The Gate-Block of the embodiment forms a penetration gate control mechanism integrating optical penetration, fog field diffusion and channel regulation through the joint action of the suppression path of the r-Gate and the enhancement path of the a-Gate, effectively enhances the dynamic adaptive ability of the Transformer model in the extreme degradation scene, and significantly improves the response stability and discrimination accuracy of the network to the target disturbed by thick fog.

[0059] Embodiment 4 As shown in Figure 10 The embodiment provides a penetration perception Transformer target detection device for a thick fog degradation scene, which comprises: A visual compensation module construction unit 10 is configured to construct a multi-scale visual compensation module; A module embedding unit 20 is configured to embed the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is configured to enhance local features in a thick fog scene. A spatial feature extraction unit 30 is configured to acquire a scene image in a thick fog scene, input the scene image into a target detection framework under thick fog, extract spatial features of the scene image, and obtain a first feature map; the target detection framework under thick fog is composed of a plurality of penetration perception backbone networks. A feature redistribution unit 40 is configured to perform feature redistribution on the first feature map by using a deformable attention mechanism to obtain a second feature map. A nonlinear transformation unit 50 is configured to perform nonlinear transformation on the second feature map to simulate global light diffusion effects and obtain a fourth feature map. A visibility mapping unit 60 is configured to calculate a fog layer transmittance matrix according to a transmittance function, the transmittance function is configured to simulate the fog layer visibility change of each pixel point of the scene image, and perform visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features. A residual connection unit 70 is configured to perform residual connection between the target region features and the scene image to obtain a target in a thick fog scene.

[0060] The penetration perception Transformer target detection device for a thick fog degradation scene of the embodiment is configured to perform the penetration perception Transformer target detection method for a thick fog degradation scene of any one of embodiments 1-3.

[0061] The experimental platform of the embodiment is a deep learning server, the deep learning server is loaded with a Linux operating system, the CPU model of the deep learning server is Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz, the GPU model is NVIDIA GeForce RTX A6000, 48GB display memory, and the memory size is 64GB. The programming language is Python, all deep learning frameworks are based on PyTorch, the development environment is PyTorch 1.10.1, CUDA 11.3, and Python 3.8.

[0062] Preparation of the data set: the proposed method is evaluated on the natural foggy day data set RTTS, the RTTS data set belongs to the target detection data set of the natural foggy day environment, and the images mainly come from traffic roads and the like. The RTTS data set contains 4322 images with labels of 5 categories, namely cars, people, buses, bicycles and motorcycles, and includes about 41000 targets. For the KITTI data set, the 2D data set thereof is used to perform a target detection task, and the 2D data set contains 7481 images including 6 categories: cars, vans, pedestrians, trams, trucks and bicycles. The MS COCO data set has 80 categories, in order to make the experimental detection objects consistent, 10,000 images in COCO 2014 val are selected, including categories such as motorcycles, cars, buses, trains, bicycles and trucks.

[0063] Table 1 Detection results of different types of targets by the method of the present application and various existing methods

[0064] The label distribution of the RTTS dataset is uneven, and different categories are similar at certain angles. The method provided by the present application overcomes these difficulties to some extent and achieves good experimental results. As shown in Table 1, the method proposed in the present application is significantly better than other methods in the table in all target categories and overall indicators. mAP@0.5 represents the average precision mean when the intersection over union threshold is 0.5. Compared with the strongest contrast baseline (FPVDNet in the table, mAP = 45.00) in the table, the mAP@0.5 of the method provided by the present application is 65.39, which is an absolute improvement of 20.39 and a relative improvement of 45.31%. The absolute improvement refers to the difference in numerical value between the two experimental data, and the relative improvement refers to the ratio obtained by dividing the absolute improvement by the smaller experimental data. From different categories, the car category is improved from 65.47 of the closest FPVDNet to 76.41, with an absolute improvement of 10.94 and a relative improvement of 16.71%; the bus category is improved from 51.53 to 55.78, with an absolute improvement of 4.25 and a relative improvement of 8.25%; the motorcycle category is improved from 32.20 to 53.62, with an absolute improvement of 21.42 and a relative improvement of 66.52%; the bicycle category is improved from 46.33 to 58.12, with an absolute improvement of 11.79 and a relative improvement of 25.45%; and the pedestrian category is improved from 46.61 to 66.92, with an absolute improvement of 20.31 and a relative improvement of 43.57%. The above results show that the method provided by the present application has significant advantages in small-scale, occluded, and / or diverse targets such as pedestrians and motorcycles, and also has robust improvements in common targets such as bicycles and cars. Overall, the target detection network of the present application provides stronger representation ability and more robust detection performance, thereby significantly improving the recall rate and / or positioning accuracy, and ultimately reflecting a substantial improvement in mAP.

[0065] Figures 7-9 is the test result of the method provided by the present application on the test set, from Figures 7-9 It can be seen that the target detection model of the present application can accurately detect targets of different scales, such as motorcycles, cars and pedestrians, in dense fog scenes.

[0066] The computer device in the embodiment can be a server, wherein the computer device comprises a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with external terminals through network connection.

[0067] The embodiment also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method for dense fog degradation scene oriented penetration perception Transformer target detection according to any one of the embodiments 1-3. It can be understood that the computer readable storage medium in the embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0068] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, device, article or method that includes a list of elements not only includes those elements, but also includes other elements not expressly listed, or inherent to such a process, device, article or method. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, device, article or method that includes the element.

[0069] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is based on the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. A penetration-aware Transformer target detection method for dense fog-degraded scenes, characterized by: include: Construct a multi-scale visual compensation module; The multi-scale visual compensation module is embedded in the Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in dense fog scenes; Acquire a scene image in a dense fog scene, input the scene image into a dense fog target detection framework, perform spatial feature extraction on the scene image, and obtain a first feature map; the dense fog target detection framework is composed of a plurality of the penetration perception backbone networks; Using a deformable attention mechanism to redistribute features of the first feature map to obtain a second feature map; Performing a nonlinear transformation on the second feature map to simulate a global illumination diffusion effect to obtain a fourth feature map; Calculating a fog layer penetration matrix according to a penetration function, wherein the penetration function is used to simulate the visibility change of the fog layer at each pixel point of the scene image, and performing visibility mapping on the fourth feature map based on the fog layer penetration matrix to obtain target area features; The target area features are residually connected with the scene image to obtain the target in the dense fog scene.

2. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 1 is characterized in that: The multi-scale visual compensation module is constructed, including: Design the s-Block module, s-se-Block module, and Gate-Block module; wherein the s-Block module is used to improve the clarity of local textures and object boundaries in the image; The s-Block module, s-se-Block module and Gate-Block module are combined into a multi-scale visual compensation module.

3. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 2 is characterized in that: The multi-scale visual compensation module is embedded in the Transformer architecture to obtain a penetration perception backbone network, including: Connect the input of the first coding layer to the input layer, and connect the output of the first coding layer to the input of the first s-Block module; Connecting the output of the first s-Block module to the input of the second block coding layer, and connecting the output of the second block coding layer to the input of the second s-Block module; Connecting the output of the second s-Block module to the input of the third coding layer, connecting the output of the third coding layer to the input of the third s-Block module, and connecting the output of the third s-Block module to the input of the s-se-Block module; Connect the output end of the s-se-Block module to the input end of the fourth block coding layer, connect the output end of the fourth block coding layer to the input end of the fourth s-Block module, connect the output end of the fourth s-Block module to the input end of the Gate-Block module, and connect the output end of the Gate-Block module to the output layer.

4. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 1 is characterized in that: The method of redistributing features of the first feature map using a deformable attention mechanism to obtain a second feature map includes: Inputting the first feature map into the first subnetwork of the deformable attention module to generate a position offset of the sampling point; Input the first feature map into the second sub-network of the deformable attention module to generate attention weights of the sampling points; The position offset of the sampling point is used to adjust the position of the sampling point to obtain a feature map after position adjustment; Based on the attention weights of the sampling points, weighted summation is performed on all sampling points of the position-adjusted feature map to obtain a second feature map.

5. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 4 is characterized in that: Inputting the first feature map into the first sub-network of the deformable attention module to generate a position offset of the sampling point includes: The position offset of the sampling point is generated according to the following formula: ； in, is the position offset of the sampling point, Conv offset is the first sub-network, which is used to predict the position offset, and X1 is the first feature map.

6. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 1 is characterized in that: The performing a nonlinear transformation on the second feature map to simulate a global illumination diffusion effect to obtain a fourth feature map includes: Convolving the second feature map with a channel convolution kernel to perform channel transformation on the second feature map to obtain a third feature map; Performing average pooling on the third feature map to obtain pooled features; The first feature map, the third feature map and the pooled feature are fused to obtain a fourth feature map.

7. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 1 is characterized in that: The calculation of the fog layer penetration matrix according to the penetration function includes: The fog layer penetration matrix is ​​calculated according to the following formula: ； ； Wherein, d(x) is the perceived depth, which is used to characterize the blur degree of each position in the dense fog scene. is the fog intensity adjustment coefficient, max is the maximum value function, I c (x) is the pixel intensity of the cth channel of the image pixel at position x, max c (I c (x)) is the maximum brightness channel value of the pixel at position x in the image, c is the brightness channel number; T(x) is the fog layer penetration matrix, exp is the exponential function, is the simulated fog concentration coefficient.

8. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 2 is characterized in that: The Gate-Block module includes an r gate and an a gate, and the output of the Gate-Block module is: ； ； ； Among them, G r (input) is the output of the r gate, is the activation function, input is the output of the fourth s-Block module, and FC is the fully connected layer; is the output of the a gate, AvgPool is the average pooling operation, FFN is the feedforward network, Output is the output of the Gate-Block module, and * is element-by-element multiplication.

9. The penetration-aware Transformer target detection method for dense fog degradation scenes according to claim 1 is characterized in that: The performing visibility mapping on the fourth feature map based on the fog layer penetration matrix to obtain target area features includes: The target region characteristics are calculated according to the following formula: ； Wherein, Y is the target area feature, FFN is the forward feedback network, T(x) is the fog layer penetration matrix, X4 is the fourth feature map, and . is a multiplication operation.

10. A penetration-aware Transformer target detection device for dense fog-degraded scenes, characterized by: include: A visual compensation module construction unit, used to construct a multi-scale visual compensation module; A module embedding unit, configured to embed the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is configured to enhance local features in dense fog scenes; a spatial feature extraction unit configured to obtain a scene image in a dense fog scene, input the scene image into a dense fog target detection framework, perform spatial feature extraction on the scene image, and obtain a first feature map; the dense fog target detection framework is composed of a plurality of the penetration sensing backbone networks; a feature redistribution unit, configured to redistribute features of the first feature map using a deformable attention mechanism to obtain a second feature map; a nonlinear transformation unit, configured to perform a nonlinear transformation on the second feature map to simulate a global illumination diffusion effect, thereby obtaining a fourth feature map; a visibility mapping unit, configured to calculate a fog layer penetration matrix based on a penetration function, wherein the penetration function is used to simulate the visibility change of the fog layer at each pixel point of the scene image, and perform visibility mapping on the fourth feature map based on the fog layer penetration matrix to obtain target area features; The residual connection unit is used to perform a residual connection between the target area feature and the scene image to obtain the target in the dense fog scene.

Images