A method and device for target detection of a penetration perception Transformer oriented to a thick fog degradation scene

Abstract

The application provides a thick fog degradation scene-oriented penetration perception Transformer target detection method and device, wherein the method comprises constructing a multi-scale visual compensation module and embedding the multi-scale visual compensation module into a Transformer architecture. A scene image is input into a thick fog target detection framework, and spatial feature extraction is performed on the scene image. A deformable attention mechanism is used to perform feature redistribution on a first feature map and nonlinear transformation on a second feature map to simulate a global light diffusion effect. A fourth feature map is subjected to visibility mapping based on a fog layer penetration rate matrix, and target region features are connected with the scene image in residual connection to obtain a target in a thick fog scene. The above method enhances the recovery capability of local details while globally modeling, and the role of the visibility mapping is to make the FFN output more focused on the target region that still retains significant features after penetration. Starting from atmospheric degradation modeling, the method can integrate penetration perception, has physical interpretability and multi-scale target perception capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a penetrating perception Transformer target detection method and device for thick fog degradation scenarios. BACKGROUND

[0002] In recent years, target detection technology has made significant progress in the field of computer vision, especially under the promotion of deep learning frameworks such as YOLO, FasterR-CNN, and DETR, efficient detection of traffic targets in clear images has been achieved. However, in actual intelligent traffic applications, especially in low-visibility extreme weather conditions such as morning fog, haze, sand, and rain and snow, traditional methods face the problem of performance degradation.

[0003] In a thick fog scenario, water vapor and particulate matter in the atmosphere can cause severe Mie scattering and Rayleigh scattering, resulting in low contrast, strong degradation, and structure loss in images. This complex degradation mechanism not only reduces image quality, but also directly damages the ability of deep learning models to distinguish key targets.

[0004] Existing target detection methods in thick fog scenarios include the following categories: (1) Image enhancement + detection two-stage method, combining classic image dehazing algorithms such as dark channel prior with target detection algorithms. Although this method improves image visual quality, the enhancement network and detection task are separated, which cannot perceive target semantic information, resulting in limited improvement in target detection performance. (2) Detection improvement network based on attention mechanism, some research attempts to introduce SE and CBAM channel attention structures in the detection model to enhance the salient target area in the degraded image. However, this method only optimizes from the feature selection perspective and cannot model the optical degradation process caused by thick fog, so it still performs unstable in extremely low visibility conditions. (3) Transformer visual modeling architecture, Vision Transformer methods such as Swin-Transformer and Deformable DETR have attracted attention due to their global perception ability. This method has certain advantages in identifying targets in thick fog scenarios. However, it still has problems in practical applications: the Transformer architecture is difficult to model the local spatial changes of fog, lacks a simulation penetration mechanism, and performs poorly in multi-scale small target recognition. (4) Methods that fuse modalities or deep priors, some methods attempt to use infrared images, depth maps, and other information to guide detection, but are limited by cost and environmental adaptability, and do not have the ability to be widely deployed in traffic scenarios.

[0005] In summary, there is an urgent need for a detection framework that models atmospheric degradation, can integrate penetrating perception, and has physical interpretability and multi-scale target perception ability. SUMMARY

[0006] To overcome the problems in the related art, the purpose of the present application is to provide a penetrating perception Transformer target detection method and device for thick fog degradation scenarios, wherein the method can fuse penetrating perception from atmospheric degradation modeling, and has physical interpretability and multi-scale target perception capability.

[0007] A penetrating perception Transformer target detection method for thick fog degradation scenarios, comprising:

[0008] Constructing a multi-scale visual compensation module;

[0009] Embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is used to enhance local features in a thick fog scenario.

[0010] Obtaining a scene image in a thick fog scenario, inputting the scene image into a target detection framework under thick fog, performing spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under thick fog is composed of a plurality of penetrating perception backbone networks;

[0011] Using a deformable attention mechanism to perform feature redistribution on the first feature map to obtain a second feature map;

[0012] Performing a nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map;

[0013] Calculating a fog layer transmission rate matrix according to a transmission rate function, the transmission rate function is used to simulate the change of fog layer visibility of each pixel point of the scene image, and performing visibility mapping on the fourth feature map based on the fog layer transmission rate matrix to obtain target region features;

[0014] Residual connecting the target region features and the scene image to obtain a target in a thick fog scenario.

[0015] In the preferred technical solution of the present application, the multi-scale visual compensation module is constructed, comprising:

[0016] Designing an s-Block module, an s-se-Block module and a Gate-Block module; wherein the s-Block module is used to improve the definition of image local texture and object boundary;

[0017] The s-Block module, the s-se-Block module and the Gate-Block module are combined to form a multi-scale visual compensation module.

[0018] In the preferable technical scheme of the present application, the multi-scale visual compensation module is embedded into the Transformer architecture to obtain a penetrating perception backbone network, which comprises:

[0019] The input end of the first block encoding layer is connected to the input layer, and the output end of the first encoding layer is connected to the input end of the first s-Block module;

[0020] The output end of the first s-Block module is connected to the input end of the second block encoding layer, and the output end of the second block encoding layer is connected to the input end of the second s-Block module;

[0021] The output end of the second s-Block module is connected to the input end of the third block encoding layer, the output end of the third block encoding layer is connected to the input end of the third s-Block module, and the output end of the third s-Block module is connected to the input end of the s-se-Block module;

[0022] The output end of the s-se-Block module is connected to the input end of the fourth block encoding layer, the output end of the fourth block encoding layer is connected to the input end of the fourth s-Block module, the output end of the fourth s-Block module is connected to the input end of the Gate-Block module, and the output end of the Gate-Block module is connected to the output layer.

[0023] In the preferable technical scheme of the present application, the deformable attention mechanism is used to perform feature redistribution on the first feature map to obtain a second feature map, which comprises:

[0024] The first feature map is input into the first subnetwork of the deformable attention module to generate a position offset of a sampling point;

[0025] The first feature map is input into the second subnetwork of the deformable attention module to generate an attention weight of the sampling point;

[0026] The position of the sampling point is adjusted using the position offset of the sampling point to obtain a position-adjusted feature map;

[0027] Based on the attention weight of the sampling point, all sampling points of the position-adjusted feature map are weighted and summed to obtain a second feature map.

[0028] In the preferable technical scheme of the present application, the first feature map is input into the first subnetwork of the deformable attention module to generate a position offset of a sampling point, which comprises:

[0029] The position offset of the sampling point is generated according to the following formula:

[0030] ;

[0031] wherein, is a position offset of a sampling point, Conv offset is a first sub-network, the first sub-network is used for predicting a position offset, X1 is the first feature map.

[0032] In the preferred technical solution of the present application, the second feature map is subjected to nonlinear transformation to simulate a global light diffusion effect, and a fourth feature map is obtained, which comprises:

[0033] The second feature map is subjected to convolution by using a channel convolution kernel to perform channel transformation on the second feature map, and a third feature map is obtained.

[0034] The third feature map is subjected to average pooling to obtain a pooled feature.

[0035] The first feature map, the third feature map and the pooled feature are fused to obtain a fourth feature map.

[0036] In the preferred technical solution of the present application, the transmittance matrix of the fog layer is calculated according to a transmittance function, which comprises:

[0037] The transmittance matrix of the fog layer is calculated according to the following formula:

[0038] ;

[0039] ;

[0040] wherein, d(x) is a perceived depth, the perceived depth is used to represent the blurring degree of each position in a dense fog scene, is a fog intensity adjustment coefficient, max is a maximum value function, I c (x) is the pixel intensity of the cth channel of the pixel point of the image at position x, max c (I c (x)) is the maximum luminance channel value of the pixel point of the image at position x, c is the luminance channel serial number; T(x) is the transmittance matrix of the fog layer, exp is an exponential function, is a simulated fog concentration coefficient.

[0041] In the preferred technical solution of the present application, the Gate-Block module comprises an r gate and an a gate, and the output of the Gate-Block module is:

[0042] ;

[0043] ;

[0044] ;

[0045] wherein, G r (input) is the output of the r gate, is an activation function, input is the output of the fourth s-Block module, FC is a full connection layer; is the output of the a gate, AvgPool is an average pooling operation, FFN is a forward feedback network, Output is the output of the Gate-Block module, and * is an element-wise multiplication.

[0046] In the preferred technical solution of the present application, the visibility mapping of the fourth feature map based on the fog layer transmittance matrix to obtain the target region feature comprises:

[0047] The target region feature is calculated according to the following formula:

[0048] ;

[0049] wherein, Y is the target region feature, FFN is a forward feedback network, T(x) is a fog layer transmittance matrix, X4 is the fourth feature map, and. is a multiplication operation.

[0050] The present application also provides a penetration perception Transformer target detection device for thick fog degradation scenes, comprising:

[0051] A visual compensation module construction unit is configured to construct a multi-scale visual compensation module.

[0052] A module embedding unit is configured to embed the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is configured to enhance local features in a thick fog scene.

[0053] A spatial feature extraction unit is configured to obtain a scene image in a thick fog scene, input the scene image into a target detection framework under thick fog, and perform spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under thick fog is composed of a plurality of penetration perception backbone networks.

[0054] A feature redistribution unit is configured to perform feature redistribution on the first feature map using a deformable attention mechanism to obtain a second feature map.

[0055] A nonlinear transformation unit is configured to perform nonlinear transformation on the second feature map to simulate a global light diffusion effect, and obtain a fourth feature map.

[0056] A visibility mapping unit is configured to calculate a fog layer transmittance matrix according to a transmittance function used to simulate the change in the visibility of the fog layer at each pixel point of the scene image, and perform visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain a target region feature.

[0057] A residual connection unit is configured to perform residual connection between the target region feature and the scene image to obtain a target in the thick fog scene.

[0058] The present application has the following advantages:

[0059] The thick fog degradation scene-oriented penetration perception Transformer target detection method provided by the present application comprises the following steps: constructing a multi-scale visual compensation module, embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network, wherein the penetration perception backbone network is used to enhance local features in a thick fog scene; obtaining a scene image in the thick fog scene, inputting the scene image into a target detection framework in the thick fog, performing spatial feature extraction on the scene image to obtain a first feature map, and the target detection framework in the thick fog is composed of a plurality of penetration perception backbone networks. On the basis of the multi-scale Transformer backbone, the ViT module is embedded in the feature layer at different resolutions, and the coupling attention mechanism is formed by combining the multi-scale visual compensation module, so as to enhance the recovery ability of local details while modeling globally. The deformable attention mechanism is used to perform feature redistribution on the first feature map to obtain a second feature map. The second feature map is subjected to nonlinear transformation to simulate the global light dispersion effect, and a fourth feature map is obtained. The nonlinear transformation comprises convolution, average pooling operation and weighted fusion, and the fourth feature map is a feature map with penetration perception capability. The fog layer transmittance is calculated according to a transmittance function used to simulate the change in the visibility of the fog layer at each pixel point of the scene image, and the fourth feature map is subjected to visibility mapping based on the fog layer transmittance to obtain a target region feature. The fog layer transmittance matrix is used to guide the feature weighting path in the FFN module, and the fog layer transmittance matrix participates in weight regulation in the forward propagation process, so as to adjust the output weight of the FFN structure. The role of the visibility mapping is to make the FFN output more focused on the target region which still retains significant features after penetration. The target region feature and the scene image are subjected to residual connection to obtain a target in the thick fog scene, and the residual connection plays a role in ensuring stable propagation of the feature. The method provided by the present application is based on atmospheric degradation modeling, can integrate penetration perception, and has material interpretability and multi-scale target perception capability. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 It is a flowchart of the thick fog degradation scene-oriented penetration perception Transformer target detection method of the present application.

[0061] Figure 2is a flowchart of the first feature map re-distribution process of the present application using a deformable attention mechanism;

[0062] Figure 3 is a structural diagram of the present application of the penetrating perception backbone network;

[0063] Figure 4 is a structural diagram of the s-Block module of the present application;

[0064] Figure 5 is a structural diagram of the s-se-Block module of the present application;

[0065] Figure 6 is a structural diagram of the Gate-Block module of the present application;

[0066] Figure 7 is the first target recognition result diagram under the thick fog scene of the present application;

[0067] Figure 8 is the second target recognition result diagram under the thick fog scene of the present application;

[0068] Figure 9 is the third target recognition result diagram under the thick fog scene of the present application;

[0069] Figure 10 is a schematic diagram of the penetrating perception Transformer target detection device for the thick fog degradation scene of the present application. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0071] Example 1

[0072] As shown in Figure 1 , the present embodiment provides a penetrating perception Transformer target detection method for a thick fog degradation scene, comprising:

[0073] S1: constructing a multi-scale visual compensation module;

[0074] S2: embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is used to enhance local features in a thick fog scene;

[0075] S3: Acquire a scene image under dense fog, input the scene image into the target detection framework under dense fog, extract spatial features from the scene image to obtain a first feature map; the target detection framework under dense fog is composed of multiple penetration sensing backbone networks;

[0076] S4: The first feature map is redistributed using a deformable attention mechanism to obtain the second feature map;

[0077] S5: Perform a nonlinear transformation on the second feature map to simulate the global illumination diffusion effect and obtain the fourth feature map;

[0078] S6: Calculate the fog penetration rate matrix based on the penetration rate function, which is used to simulate the fog visibility change of each pixel in the scene image. Perform visibility mapping on the fourth feature map based on the fog penetration rate matrix to obtain the target area features.

[0079] S7: Perform residual connection between the target region features and the scene image to obtain the target in the dense fog scene.

[0080] like Figure 3 As shown, the scene image is input into the first s-Block module of the first penetration perception backbone network of the target detection framework under dense fog, namely PenetrateViT. The scene image is subjected to 3×3 depthwise separable convolution and 1×1 depthwise separable convolution respectively. The results of the two depthwise separable convolutions are then concatenated, and the concatenated result is input into the batch normalized neural network to extract the first feature map.

[0081] The step of extracting spatial features from the scene image to obtain a first feature map includes:

[0082] The first feature map is extracted from the scene image using depthwise separable convolution according to the following formula:

[0083] ;

[0084] Where X1 is the first feature map, BN is the batch normalized neural network, and DWConv 3×3 For a depthwise separable convolution of size 3×3, DWConv 1×1 The method uses a 1×1 depthwise separable convolution, where X is the scene image. This embodiment extracts spatial features from the scene image, enabling lightweight spatial awareness modeling while preserving the original texture response.

[0085] The construction of the multi-scale visual compensation module includes:

[0086] S11: design s-Block module, s-se-Block module and Gate-Block module; wherein the s-Block module is used to improve the clarity of local texture and object boundary of the image;

[0087] S12: compose s-Block module, s-se-Block module and Gate-Block module into a multi-scale visual compensation module.

[0088] The s-Block module simulates non-uniform spatial deformation through a deformable attention mechanism, simulates optical degradation guidance through a transmittance function, and under the mechanism of joint modeling of local features and global features, strengthens the structural penetration perception ability of the Transformer network under heavy fog, and improves the detection performance of the model on distant, small-scale and edge blurred targets.

[0089] The deformable attention mechanism can only allocate attention to key pixels, greatly reducing the amount of calculation and having stronger robustness to deformation and scale changes. The deformable attention module selects 4-8 sampling points, the offset varies with the input data, the receptive field can be deformed, and different scale features can be sampled.

[0090] As shown in Figure 2 , the deformable attention mechanism is used to redistribute the features of the first feature map to obtain a second feature map, comprising:

[0091] S41: input the first feature map into a first subnetwork of the deformable attention module to generate a position offset of the sampling point;

[0092] S42: input the first feature map into a second subnetwork of the deformable attention module to generate an attention weight of the sampling point;

[0093] S43: adjust the position of the sampling point using the position offset of the sampling point to obtain a position-adjusted feature map;

[0094] S44: based on the attention weight of the sampling point, weighted sum all sampling points of the position-adjusted feature map to obtain a second feature map.

[0095] The calculation process of the deformable attention module is as follows:

[0096] ;

[0097] ;

[0098] wherein, is the position offset of the sampling point, Conv offsetis a first sub-network for predicting the position offset of each sampling point, X1 is the first feature map, W is the attention weight of the sampling point, Conv weight is a second sub-network. The first sub-network Conv offset is used to predict the position offset of each sampling point, and the output dimension is Kx2, which represents the displacement of the K sampling points in the x-axis direction and the y-axis direction. The second sub-network Conv weight is used to predict the attention weight w of each offset sampling point k , and the output dimension is K, which represents the attention weight of the K offset sampling points. The parameters of the first sub-network and the second sub-network are automatically learned from data by a back propagation algorithm, i.e. the parameters of the two sub-networks are trainable.

[0099] Preferably, the attention weight of each offset sampling point is normalized by using a normalized exponential function, so that the sum of the attention weights of the K offset sampling points is 1. The normalized attention weight of the kth offset sampling point is calculated according to the following formula:

[0100] ;

[0101] wherein, is the normalized attention weight of the kth offset sampling point, exp is the exponential function, w k is the attention weight of the kth offset sampling point, w j is the attention weight of the jth offset sampling point, K is the total number of offset sampling points, and j is the serial number of the offset sampling point. The deformable attention mechanism can improve the modeling ability of boundary distortion and target offset in heavy fog.

[0102] After the K sampling points are adjusted in position, K offset sampling points are obtained. Based on the position offset of the sampling points and the attention weight of the sampling points, the second feature map is calculated using the following formula:

[0103] ;

[0104] ;

[0105] wherein, X2 is the second feature map, DeformAttn is the output of the deformable attention module, X1 is the first feature map, is the position offset of the sampling point, is a matrix; w k is the attention weight of the kth offset sampling point, p k is the coordinate of the kth sampling point in the first feature map, is the position offset of the kth sampling point in the first feature map,. is a multiplication operation. p is used to adjustk to update the coordinates of the sampling points in the first feature map, to obtain the coordinates of the offset sampling points in the first feature map, and the values corresponding to all the coordinates of the offset sampling points form a position-adjusted feature map. Alternatively, in the above formula, is replaced by .

[0106] The deformable attention mechanism can simulate feature drift caused by optical scattering and refraction in a thick fog environment. Compared with an ordinary attention mechanism, the deformable attention mechanism is more suitable for modeling non-rigid changes caused by the shape of a target in a thick fog, and improves the spatial adaptability of the model in an unstructured scene.

[0107] In order to enhance the interpretability of the object, a transmittance function is designed to simulate the visibility change of each pixel point in the image in the fog layer. The transmittance matrix of the fog layer is calculated according to the transmittance function, and the transmittance matrix of the fog layer participates in the weight regulation in the forward propagation process. The visibility mapping makes the output of the FFN more focused on the target area which still retains significant features after penetration. The target area feature Y is connected with the scene image X in residual connection, that is, Z = X + Y, Z is the target in the thick fog scene, so as to ensure the stable propagation of the feature.

[0108] ​The thick fog degradation scene-oriented penetration perception Transformer target detection method provided by the embodiment comprises: constructing a multi-scale visual compensation module, embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene. A scene image under a thick fog scene is acquired, the scene image is input into a target detection framework under a thick fog, spatial feature extraction is performed on the scene image to obtain a first feature map; the target detection framework under the thick fog is composed of a plurality of penetration perception backbone networks. On the basis of a multi-scale Transformer backbone, a ViT module is embedded in a feature layer of different resolutions, and a coupled attention mechanism is formed in combination with the multi-scale visual compensation module to enhance the recovery capability of local details while globally modeling. A deformable attention mechanism is used to perform feature redistribution on the first feature map to obtain a second feature map; a nonlinear transformation is performed on the second feature map to simulate a global light dispersion effect to obtain a fourth feature map. The nonlinear transformation comprises convolution, average pooling operation and weighted fusion, and the fourth feature map is a feature map with penetration perception capability. The fog layer penetration rate is calculated according to a penetration rate function, the penetration rate function is used to simulate the fog layer visibility change of each pixel point of the scene image, the fourth feature map is subjected to visibility mapping based on the fog layer penetration rate to obtain target region features. The fog layer penetration rate matrix is used to guide the feature weighting path in the FFN module, and the fog layer penetration rate matrix participates in weight regulation in the forward propagation process, so that the output weight of the FFN structure is adjusted. The role of the visibility mapping is to make the FFN output more focused on the target region that still retains significant features after penetration. The target region features are connected with the scene image in residual connection to obtain a target under a thick fog scene, and the residual connection plays a role in ensuring stable feature propagation. The method provided by the present application starts from atmospheric degradation modeling, can integrate penetration perception, and has material interpretability and multi-scale target perception capability.

[0109] Embodiment 2

[0110] The embodiment provides a thick fog degradation scene-oriented penetration perception Transformer target detection method, and the embodiment describes the differences from the embodiment 1 on the basis of the embodiment 1. The method comprises the following steps:

[0111] S1: constructing a multi-scale visual compensation module;

[0112] S2: embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene.

[0113] S3: acquire a scene image in a thick fog scene, input the scene image into a target detection framework under thick fog, perform spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under thick fog is composed of a plurality of the penetrating perception backbone networks;

[0114] S4: feature redistribution is performed on the first feature map by using a deformable attention mechanism to obtain a second feature map;

[0115] S5: nonlinear transformation is performed on the second feature map to simulate a global light diffusion effect, and a fourth feature map is obtained;

[0116] S6: a fog layer transmittance matrix is calculated according to a transmittance function, the transmittance function is used to simulate the fog layer visibility change of each pixel point of the scene image, and the fourth feature map is mapped based on the fog layer transmittance matrix to obtain a target region feature;

[0117] S7: residual connection is performed on the target region feature and the scene image to obtain a target in a thick fog scene.

[0118] The multi-scale visual compensation module comprises:

[0119] S11: an s-Block module, an s-se-Block module and a Gate-Block module are designed; wherein the s-Block module is used to improve the definition of local texture and object boundary of an image;

[0120] S12: the s-Block module, the s-se-Block module and the Gate-Block module are combined to form a multi-scale visual compensation module.

[0121] Figure 4 The structure diagram of the s-Block module, which is the core visual compensation structure in the application, aims to alleviate the problems of local texture blurring and structure boundary disappearance in a thick fog environment. The s-Block module is embedded in the backbone of the Transformer in a residual manner, and by introducing a deformable attention mechanism, an adaptive convolution combination and a shallow feature fusion mechanism, the local non-uniform light scattering characteristics in a thick fog environment can be simulated, and the physical enhanced perception of the target edge region can be realized. The processing of the s-Block module on data includes the following steps: (1) performing spatial feature extraction on a scene image in a thick fog scene to obtain a first feature map; (2) inputting the first feature map into a deformable attention module to obtain a second feature map; (3) performing nonlinear transformation on the second feature map to simulate a global light diffusion effect, and obtaining a fourth feature map.

[0122] The second feature map is nonlinearly transformed to simulate a global light diffusion effect, to obtain a fourth feature map, including:

[0123] S51: A channel convolution kernel is used to convolve the second feature map to perform channel transformation on the second feature map, to obtain a third feature map;

[0124] S52: The third feature map is average-pooled to obtain a pooled feature;

[0125] S53: The first feature map, the third feature map, and the pooled feature are fused to obtain a fourth feature map.

[0126] The fourth feature map is calculated according to the following formula:

[0127] ;

[0128] ;

[0129] ;

[0130] wherein X3 is the third feature map, Conv 1×1 is a convolution operation with a size of 1*1, X2 is the second feature map; X avg is the pooled feature, AvgPool is average-pooling, X4 is the fourth feature map, is a fusion coefficient.

[0131] The second feature map is channel-transformed using a 1*1 convolution kernel, and the 1*1 convolution kernel is used to change the number of channels of the feature map under the premise that the image width W and height H are unchanged. The global average pooling can simulate the global light diffusion effect in the atmospheric degradation image. The global light diffusion effect is a macroscopic phenomenon of light re-emission after multiple scattering in the medium. The first feature map, the third feature map, and the pooled feature are fused by using weighted summation, and the fusion coefficient is adjusted, so that the fourth feature map has a component of low-frequency diffuse reflection of global light, and the modeling ability of the model for background bias is enhanced.

[0132] The transmittance matrix of the fog layer is calculated according to the transmittance function, including:

[0133] The transmittance matrix of the fog layer is calculated according to the following formula:

[0134] ;

[0135] ;

[0136] wherein d(x) is a perceived depth, and the perceived depth is used to represent the blurring degree of each position in the dense fog scene, is a fog intensity adjustment coefficient, max is a max function, I c (x) is a pixel intensity of a pixel at position x in the image in the cth channel, max c c (x) is a maximum luminance channel value of a pixel at position x in the image. c is a luminance channel number; T(x) is the fog layer transmittance matrix, exp is an exponential function, is a simulated fog density coefficient,. is a multiplication operation.

[0137] The maximum value of a pixel at position x, for example, the rowth row and the colth column in the RGB three channels of the scene image is calculated, wherein row≤H, col≤W, H is the height of the scene image, and W is the width of the scene image. The perceived depth d(x) represents the degree of blurring of the scene image at position x.

[0138] In order to enhance the interpretability of the object, the transmittance function is designed to simulate the visibility change of each pixel in the image in the fog layer. The fog intensity adjustment coefficient is negatively correlated with the concentration of fog and haze in the environment, that is, the higher the concentration of fog and haze in the environment, the larger the fog intensity adjustment coefficient . The fog layer transmittance matrix T(x) is used to guide the feature weighting path in the FFN module, and the fog layer transmittance matrix corresponds to a transmittance map which participates in weight regulation in the forward propagation process, so as to adjust the output weight of the FFN structure. The FFN structure is a forward feedback network, which has the characteristics of one-way data flow, that is, data is input from the input layer, passes through the hidden layer, and is output from the output layer, and the neurons in the same layer are fully connected.

[0139] The visibility mapping is performed on the fourth feature map based on the fog layer transmittance matrix to obtain a target region feature, comprising:

[0140] The target region feature is calculated according to the following formula:

[0141] ;

[0142] Wherein Y is the target region feature, FFN is a forward feedback network, T(x) is a fog layer transmittance matrix, X4 is the fourth feature map, and. is a multiplication operation.

[0143] The visibility mapping based on physical degradation makes the output of the FFN more focused on the target region which still retains significant features after penetration, and the target region feature is connected with the scene image in residual connection, so as to ensure stable propagation of the feature. The following formula is used to calculate the target in the thick fog scene: Z=X+Y, wherein X is the scene image, Y is the target region feature, and Z is the target in the thick fog scene. ​

[0144] The s-Block of the embodiment simulates non-uniform spatial deformation through a deformable attention mechanism, simulates optical degradation guidance through a transmittance function, and under the mechanism of local-global joint modeling, strengthens the structural penetration perception ability of the Transformer network in thick fog, and improves the detection performance of the model on distant, small-scale and edge blurred targets.

[0145] Embodiment 3

[0146] The embodiment provides a penetration perception Transformer target detection method for thick fog degradation scenes. The embodiment describes the differences between the embodiment 1 and the embodiment 2 on the basis of the embodiment 1. The method comprises:

[0147] S1: constructing a multi-scale visual compensation module;

[0148] S2: embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetration perception backbone network; wherein the penetration perception backbone network is used to enhance local features in a thick fog scene;

[0149] S3: obtaining a scene image in a thick fog scene, inputting the scene image into a target detection framework under thick fog, performing spatial feature extraction on the scene image to obtain a first feature map; the target detection framework under thick fog is composed of a plurality of penetration perception backbone networks;

[0150] S4: performing feature redistribution on the first feature map by using a deformable attention mechanism to obtain a second feature map;

[0151] S5: performing nonlinear transformation on the second feature map to simulate global light diffusion effect to obtain a fourth feature map;

[0152] S6: calculating a fog layer transmittance matrix according to a transmittance function, the transmittance function is used to simulate the visibility change of each pixel point of the scene image, performing visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features;

[0153] S7: performing residual connection on the target region features and the scene image to obtain a target in a thick fog scene.

[0154] The embedding of the multi-scale visual compensation module into the Transformer architecture to obtain the penetration perception backbone network comprises:

[0155] S21: connecting an input end of a first encoding layer to an input layer and connecting an output end of the first encoding layer to an input end of a first s-Block module;

[0156] S22: connecting the output end of the first s-Block module to the input end of a second block encoding layer, and connecting the output end of the second block encoding layer to the input end of a second s-Block module;

[0157] S23: connecting the output end of the second s-Block module to the input end of a third block encoding layer, and connecting the output end of the third block encoding layer to the input end of a third s-Block module, and connecting the output end of the third s-Block module to the input end of the s-se-Block module;

[0158] S24: connecting the output end of the s-se-Block module to the input end of a fourth block encoding layer, and connecting the output end of the fourth block encoding layer to the input end of a fourth s-Block module, and connecting the output end of the fourth s-Block module to the input end of the Gate-Block module, and connecting the output end of the Gate-Block module to the output layer.

[0159] The structure of the PenetrateViT is shown in Figure 3 The target detection framework under thick fog includes a backbone network and a neck network, the backbone network is composed of multiple PenetrateViT, the Transformer backbone of the PenetrateViT is unfolded in a hierarchical structure, and the input scene image is divided into N = HW / P patch tokens after passing through the first block encoding layer. 2

[0160] Figure 5 The structure diagram of the s-se-Block module is shown in the figure, the s-se-Block module is introduced on the basis of the s-Block module, and the SE attention mechanism is introduced to enhance the dynamic response ability in the channel dimension. The SE attention mechanism can simulate the significant changes between channels caused by the difference in fog particle concentration in the thick fog environment, so as to enhance the target region in the feature dimension.

[0161] Compared with the s-Block module, the core improvement of the s-se-Block module is only that the input features are sent into the SE module before being sent into the backbone convolution path, and participate in the final residual fusion after extracting the channel perception weight. The overall residual output is expressed as:

[0162]

[0163] Where, Temp s-se-Block is the output of the s-se-Block module, FFN is a forward feedback network, is the penetration rate matrix calculated by the penetration rate function when the input is the output of the third s-Block,​​ DeformAttn is deformable attention, BN is batch normalization network, f denotes a mapping function, the mapping function f is a splicing result of a 3x3 depth separable convolution and a 1x1 depth separable convolution, SE denotes a squeeze-and-excitation mechanism, Figure 5 Y' in the formula is .

[0164] After the SE mechanism is introduced, the s-se-Block can not only cope with spatial blur degradation, but also dynamically model the channel importance offset, which is a physical phenomenon of non-uniformity of the fog layer, that is, the brightness of the blue channel of the scene image in the dense scene is significantly lower than that of the red and green channels.

[0165] The Gate-Block module is used to simulate the dynamic penetration adjustment mechanism of image feature information in a dense fog environment. The core idea is to select the feature information according to the penetration difficulty of different channels and regions, so as to retain and enhance part of the feature information. The structure of the Gate-Block module is as shown in Figure 6 The Gate-Block module includes two substructures, which are a residual gating unit r-Gate and an enhanced gating unit a-Gate. The residual gating unit r-Gate is used for information filtering, and the enhanced gating unit a-Gate is used for energy compensation, thereby constructing a feature controller with active selection of penetration characteristics.

[0166] The input feature input of the Gate-Block module is processed after the Gate-Block module, and the process of obtaining the output feature of the input is as follows:

[0167] The input feature input of the Gate-Block module is processed after the Gate-Block module, and the process of obtaining the output feature of the input is as follows:

[0168] ;

[0169] wherein FC is a fully connected layer, is an activation function such as a Sigmoid function, is the output of the residual gating unit r-Gate structure, that is, the output of the r gate, and input is the output of the fourth s-Block module.

[0170] The residual gating unit r-Gate structure is embedded with two nonlinear layers. The first nonlinear layer obtains an initial penetration control amount through FC, and the second nonlinear layer fuses the input to obtain a feature map. The residual gating unit r-Gate structure plays a role in simulating information suppression of an image disturbed under dense fog conditions.

[0171] The enhancement gating unit a-Gate takes the output of the residual gating unit r-Gate as input and plays the role of guiding the features to concentrate and compensate the global fog sensing area.

[0172] ;

[0173] in, To enhance the output of the gate a-Gate, i.e., the output of gate a, AvgPool performs average pooling. The activation function is FFN, which is a feedforward network. This is the output of the residual gating unit r-Gate structure, i.e., the output of the r-gate.

[0174] The final merged output of the Gate-Block module is:

[0175] ;

[0176] Output is the output of the Gate-Block module, and * indicates element-wise multiplication.

[0177] The function of the residual gated unit r-Gate is to perform local penetration occlusion modeling, using G... r (input) Simulates the penetration suppression degree of image features across different channels in dense fog regions. Its output is a local penetration response map, which is interpreted as the difference in light penetration ability across various channels, such as color and texture channels. The enhancement gating unit a-Gate functions as a global fog field adjustment model. a (G r (input)) It integrates residual information and feedforward enhancement path, and simulates the suppressive modulation of the fog field on the overall image brightness and texture distribution through global average pooling, so as to realize the modeling compensation of fog field diffuse illumination disturbance.

[0178] In this embodiment, Gate-Block works together through the suppression path of r-Gate and the enhancement path of a-Gate to form a three-in-one penetration gating mechanism of optical penetration, fog field diffusion and channel control. This effectively enhances the dynamic adaptive capability of the Transformer model in extreme degradation scenarios and significantly improves the network's response stability and discrimination accuracy to dense fog disturbance targets.

[0179] Example 4

[0180] like Figure 10 As shown, this embodiment provides a penetration-sensing Transformer target detection device for dense fog degradation scenarios, including:

[0181] The visual compensation module construction unit 10 is configured to construct a multi-scale visual compensation module.

[0182] The module embedding unit 20 is configured to embed the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is configured to enhance local features in a heavy fog scene.

[0183] The spatial feature extraction unit 30 is configured to obtain a scene image in a heavy fog scene, input the scene image into a heavy fog target detection framework composed of a plurality of penetrating perception backbone networks, and perform spatial feature extraction on the scene image to obtain a first feature map.

[0184] The feature redistribution unit 40 is configured to perform feature redistribution on the first feature map using a deformable attention mechanism to obtain a second feature map.

[0185] The nonlinear transformation unit 50 is configured to perform nonlinear transformation on the second feature map to simulate a global light diffusion effect, and obtain a fourth feature map.

[0186] The visibility mapping unit 60 is configured to calculate a fog layer transmittance matrix according to a transmittance function configured to simulate the change in fog layer visibility of each pixel point of the scene image, and perform visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features.

[0187] The residual connection unit 70 is configured to perform residual connection between the target region features and the scene image to obtain a target in a heavy fog scene.

[0188] The penetrating perception Transformer target detection device for heavy fog degradation scenes of the embodiment is configured to perform the penetrating perception Transformer target detection method for heavy fog degradation scenes of any one of embodiments 1-3.

[0189] The experimental platform of the embodiment is a deep learning server, which is loaded with a Linux operating system, the CPU model of the deep learning server is Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz, the GPU model is NVIDIA GeForce RTX A6000 with 48GB video memory, and the memory size is 64GB. The programming language is Python, all deep learning frameworks are based on PyTorch, the development environment is PyTorch 1.10.1, CUDA 11.3, and Python 3.8.

[0190] Preparation of the dataset: The proposed method is evaluated on the natural foggy dataset RTTS, which is a target detection dataset in natural foggy environment, and the images are mainly from traffic road scenes. The RTTS dataset contains 4322 images with labels of 5 categories, i.e., car, person, bus, bicycle and motorcycle, including about 41000 targets. For the KITTI dataset, the 2D dataset thereof is used to perform the target detection task, and the 2D dataset contains 7481 images including 6 categories: car, van, pedestrian, tram, truck and bicycle. The MS COCO dataset has 80 categories, and in order to make the experimental detection objects consistent, 10,000 images in COCO 2014 val are selected, including categories such as motorcycle, car, bus, train, bicycle and truck.

[0191] Table 1. Detection results of different types of targets by the method of the present application and various existing methods

[0192]

[0193] The label distribution in the RTTS dataset is uneven, with different categories showing similarity from certain perspectives. The method provided in this invention overcomes these difficulties to some extent and achieves better experimental results. As shown in Table 1, the method proposed in this invention significantly outperforms other methods in the table across all target categories and overall metrics. mAP@0.5 represents the mean accuracy when the intersection-union ratio threshold is 0.5. Compared to the strongest baseline (FPVDNet in the table, mAP = 45.00), the method provided in this invention achieves mAP@0.5 of 65.39, representing an absolute improvement of 20.39 and a relative improvement of 45.31%. The absolute improvement refers to the numerical difference between two experimental data points, while the relative improvement is the ratio obtained by dividing the absolute improvement by the smaller experimental data point. Analyzing different categories, the performance of the vehicle category improved from 65.47 to 76.41 (the closest FPVDNet), an absolute improvement of 10.94 and a relative improvement of 16.71%; the bus category improved from 51.53 to 55.78, an absolute improvement of 4.25 and a relative improvement of 8.25%; the motorcycle category improved from 32.20 to 53.62, an absolute improvement of 21.42 and a relative improvement of 66.52%; the bicycle category improved from 46.33 to 58.12, an absolute improvement of 11.79 and a relative improvement of 25.45%; and the pedestrian category improved from 46.61 to 66.92, an absolute improvement of 20.31 and a relative improvement of 43.57%. These results demonstrate that the method provided by this invention has significant advantages for targets such as pedestrians and motorcycles that are small in scale, occluded, and / or have diverse shapes, while also showing robust improvements for more common targets such as bicycles and vehicles. Overall, the target detection network of this invention brings stronger characterization capabilities and more robust detection performance, thereby significantly improving recall and / or localization accuracy, ultimately reflected in a substantial increase in mAP.

[0194] Figures 7-9 These are the test results of the method provided by this invention on the test set, from Figures 7-9 It is understood that the target detection model of the present invention can accurately detect targets of different scales, such as motorcycles, cars and pedestrians, in dense fog scenes.

[0195] This embodiment also provides a computer device, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communication with external terminals via a network connection.

[0196] The embodiment also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method for dense fog degradation scene oriented penetration perception Transformer target detection according to any one of the embodiments 1-3. It can be understood that the computer readable storage medium in the embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0197] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, device, article or method that includes a list of elements not only includes those elements, but also includes other elements not expressly listed, or inherent to such a process, device, article or method. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, device, article or method that includes the element.

[0198] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is based on the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. A method for target detection of a penetrating perception Transformer oriented to a thick fog degradation scene, characterized in that, The method comprises the following steps: constructing a multi-scale visual compensation module; embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is used to enhance local features in a heavy fog scene; obtaining a scene image in a heavy fog scene, inputting the scene image into a heavy fog target detection framework, performing spatial feature extraction on the scene image to obtain a first feature map; the heavy fog target detection framework is composed of a plurality of penetrating perception backbone networks; using a deformable attention mechanism to perform feature redistribution on the first feature map to obtain a second feature map; performing a nonlinear transformation on the second feature map to simulate a global light diffusion effect to obtain a fourth feature map; calculating a fog layer transmittance matrix according to a transmittance function, the transmittance function is used to simulate the change of fog layer visibility of each pixel point of the scene image, and performing visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features; performing residual connection on the target region features and the scene image to obtain a target in a heavy fog scene; The method of constructing a multi-scale visual compensation module comprises: designing an s-Block module, an s-se-Block module and a Gate-Block module; wherein the s-Block module is used to improve the definition of image local texture and object boundary; the s-Block module, the s-se-Block module and the Gate-Block module are combined to form a multi-scale visual compensation module; The method of embedding the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network comprises: connecting the input end of the first block encoding layer to the input layer, and connecting the output end of the first encoding layer to the input end of the first s-Block module; connecting the output end of the first s-Block module to the input end of the second block encoding layer, and connecting the output end of the second block encoding layer to the input end of the second s-Block module; connecting the output end of the second s-Block module to the input end of the third block encoding layer, and connecting the output end of the third block encoding layer to the input end of the third s-Block module, and connecting the output end of the third s-Block module to the input end of the s-se-Block module; connecting the output end of the s-se-Block module to the input end of the fourth block encoding layer, and connecting the output end of the fourth block encoding layer to the input end of the fourth s-Block module, and connecting the output end of the fourth s-Block module to the input end of the Gate-Block module, and connecting the output end of the Gate-Block module to the output layer; The method of performing visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features comprises: calculating the target region features according to the following formula: ； wherein Y is the target region features, FFN is a forward feedback network, T(x) is a fog layer transmittance matrix, X4 is the fourth feature map, and. is a multiplication operation; The processing of data by the s-Block module comprises the following steps: (1) spatial feature extraction is performed on a scene image in a dense fog scene to obtain a first feature map; (2) the first feature map is input into a deformable attention module to obtain a second feature map; and (3) a nonlinear transformation is performed on the second feature map to simulate a global light diffusion effect, thereby obtaining a fourth feature map; The s-se-Block module is introduced by introducing an SE attention mechanism on the basis of the s-Block module, and the input feature is simultaneously input into the SE module before being sent into the main convolution path, and participates in the final residual fusion after extracting the channel perception weight, so as to enhance the dynamic response capability in the channel dimension, and the SE attention mechanism simulates the significant change between channels caused by the difference in the concentration distribution of fog particles in the dense fog environment; The Gate-Block module comprises an r gate and an a gate, and the output of the Gate-Block module is: ； ； ； wherein G r is the output of the r gate, is an activation function, input is the output of the fourth s-Block module, and FC is a fully connected layer; is the output of the a gate, AvgPool is an average pooling operation, FFN is a feed-forward network, Output is the output of the Gate-Block module, and * is an element-wise multiplication. The r gate is a residual gate unit r-Gate, and the a gate is an enhancement gate unit a-Gate; the residual gate unit r-Gate is embedded with two nonlinear layers, the first nonlinear layer obtains an initial penetration control amount through FC, and the second nonlinear layer fuses the input to obtain a feature map; the enhancement gate unit a-Gate takes the output of the residual gate unit r-Gate as input, and the enhancement gate unit a-Gate plays a role in guiding the features to concentrate compensation in the global fog sensitive area; the input feature input of the Gate-Block module is first subjected to the r-Gate structure, and the enhancement gate unit a-Gate takes the output of the residual gate unit r-Gate as input.

2. The dense fog degraded scene oriented penetration-aware Transformer object detection method of claim 1, wherein, The first feature map is input into a first subnetwork of the deformable attention module to generate a position offset of a sampling point; The first feature map is input into a second subnetwork of the deformable attention module to generate an attention weight of the sampling point; The position of the sampling point is adjusted by using the position offset of the sampling point to obtain a position-adjusted feature map; Based on the attention weight of the sampling point, all sampling points of the position-adjusted feature map are weighted and summed to obtain a second feature map. The first feature map is input into a first subnetwork of the deformable attention module to generate a position offset of a sampling point; 3. The dense fog degraded scene oriented penetration-aware Transformer object detection method of claim 2, wherein, The position offset of the sampling point is generated according to the following formula: The second feature map is subjected to a nonlinear transformation to simulate a global light diffusion effect, thereby obtaining a fourth feature map, which comprises the following steps: ； wherein, is a position offset of a sampling point, Conv offset is a first sub-network, the first sub-network being configured to predict a position offset, X1 being the first feature map.

4. The dense fog degraded scene oriented penetration-aware Transformer object detection method of claim 1, wherein, The second feature map is convolved by using a channel convolution kernel to perform channel transformation on the second feature map, thereby obtaining a third feature map; The third feature map is subjected to average pooling to obtain a pooled feature; The first feature map, the third feature map and the pooled feature are fused to obtain a fourth feature map. The fog layer penetration rate matrix is calculated according to the penetration rate function, which comprises the following steps:

5. The dense fog degraded scene oriented penetration-aware Transformer object detection method of claim 1, wherein, The fog layer penetration rate matrix is calculated according to the following formula: ​ ； ； wherein d(x) is a perception depth, the perception depth being used to represent a blur degree of each position in the dense fog scene, is a fog intensity adjustment coefficient, max is a maximum value function, I c (x) is a pixel intensity of a pixel point of a cth channel of the image at a position x, max c (I c (x)) is a maximum luminance channel value of the pixel point of the image at the position x, c is a luminance channel serial number; T(x) is the fog layer transmittance matrix, exp is an exponential function, is a simulated fog concentration coefficient.

6. A dense fog degradation scene-oriented penetration-aware Transformer target detection apparatus, characterized in that, The device is used for implementing the penetrating perception Transformer target detection method for thick fog degradation scenes according to any one of claims 1-5, and the device comprises: a visual compensation module construction unit configured to construct a multi-scale visual compensation module; a module embedding unit configured to embed the multi-scale visual compensation module into a Transformer architecture to obtain a penetrating perception backbone network; wherein the penetrating perception backbone network is configured to enhance local features in a thick fog scene; a spatial feature extraction unit configured to obtain a scene image in a thick fog scene, input the scene image into a target detection framework under thick fog, perform spatial feature extraction on the scene image, and obtain a first feature map; the target detection framework under thick fog is composed of a plurality of penetrating perception backbone networks; a feature redistribution unit configured to perform feature redistribution on the first feature map by using a deformable attention mechanism to obtain a second feature map; a nonlinear transformation unit configured to perform nonlinear transformation on the second feature map to simulate a global light diffusion effect, and obtain a fourth feature map; a visibility mapping unit configured to calculate a fog layer transmittance matrix according to a transmittance function, the transmittance function being configured to simulate the visibility change of each pixel point of the scene image, and perform visibility mapping on the fourth feature map based on the fog layer transmittance matrix to obtain target region features; a residual connection unit configured to perform residual connection on the target region features and the scene image to obtain a target in a thick fog scene.

Images