Traffic target identifying and positioning method based on foggy day scene

Through the migration generation model and target recognition and positioning model of the atmospheric light scattering prior, the problem of low target recognition accuracy of autonomous driving vehicles in foggy scenes is solved, the accuracy and speed of traffic target recognition in foggy days is improved, and the autonomous driving is enhanced System perception ability.

CN119049009APending Publication Date: 2024-11-29SHIJIAZHUANG TIEDAO UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410921230.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The target recognition and positioning accuracy of autonomous vehicles in foggy scenes is not high, resulting in a degradation of the performance of the perception system and making it difficult to identify road environments and long-distance objects.

Method used

The AtmoGAN Haze dataset is constructed using a migration generation model of atmospheric light scattering priors. The foggy-free image and foggy-containing image are converted through an adversarial generation network to generate high-quality foggy scene images, and combined with the target recognition and positioning model for identification and positioning.

Benefits of technology

It improves the recognition accuracy and speed of traffic targets in foggy scenes, solves the problems of missing data details, reduced contrast and color distortion, and enhances the perception ability of the autonomous driving system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049009A_ABST
    Figure CN119049009A_ABST
Patent Text Reader

Abstract

The invention discloses a traffic target identification and positioning method based on a foggy day scene. The method comprises the following steps: S01, processing an obtained original fog-free image and an original fog-containing image by using an atmospheric light scattering prior migration generation model to construct an AtmoGAN Haze data set; and S02, inputting the constructed AtmoGAN Haze data set into a target identification and positioning model to obtain a foggy day target identification and positioning result. According to the invention, on the basis of ensuring the detection speed, the problem that the traffic target is difficult to identify due to the conditions of target detail missing, contrast reduction, color distortion and the like in the foggy day scene can be effectively solved, and the precision and speed of foggy day target identification and positioning are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle detection, and in particular to a method for identifying and positioning a traffic target based on a foggy scene. Background Art

[0002] With the rapid development of deep learning technology, autonomous driving systems have achieved remarkable results. However, in the face of severe weather conditions, especially in complex traffic scenarios such as foggy days, the perception systems of autonomous driving vehicles often face huge challenges. In foggy conditions, particles in the atmosphere, such as tiny water droplets or ice crystals, cause the scattering effect of light, resulting in blurred targets, weakened image contrast, and missing details when the camera is shooting. This directly affects the effective recognition and precise positioning of the road environment, pedestrians, and surrounding vehicles by autonomous driving vehicles, especially the reliable recognition of distant objects becomes extremely difficult.

[0003] In practical applications, foggy weather not only poses a potential threat to driving safety, but also greatly reduces the performance of autonomous driving systems in complex traffic environments. Therefore, how to improve the recognition and positioning accuracy of autonomous driving systems in foggy weather scenarios and enhance the perception and decision-making capabilities of autonomous driving has become the focus and difficulty of research in this field. Although the current traffic target recognition and positioning model performs well in clear weather, its effect is not satisfactory under adverse weather conditions such as foggy days. The adverse conditions cause the clarity and quality of camera shooting to be damaged, making the recognition and positioning of the target blurred and inaccurate. Summary of the invention

[0004] In order to solve the above problems, and to solve the problem that the recognition accuracy of the target recognition and positioning method of the autonomous driving vehicle in the foggy scene is not high, the present invention provides a traffic target recognition and positioning method based on the foggy scene.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] Traffic target recognition and positioning method based on foggy scene, including:

[0007] S01: Use the migration generative model of atmospheric light scattering prior to process the acquired original haze-free images and original haze images to construct the AtmoGAN Haze dataset;

[0008] The migration generation model includes an atmospheric scattering model and an adversarial generation network, wherein the adversarial generation network includes two generators (G, F) and two discriminators ( , );

[0009] The generator G includes an encoder, a residual network and a decoder. The encoder extracts features from the original fog-free image and converts it into a feature map through the residual network. The atmospheric scattering model generates atmospheric parameters for encoding and fusion with the feature map. The target foggy image is then output after deconvolution DeConv (DeConvolution) by the decoder. Cycle consistency loss is introduced so that the content of the output target foggy image is consistent with the input image.

[0010] The generator F includes an encoder, a residual network and a decoder. The encoder extracts features from the original foggy image, converts it into a feature map through the residual network, and then outputs a target fog-free image after deconvolution through the decoder. Cycle consistency loss is introduced so that the content of the output target fog-free image is consistent with the input image.

[0011] At the same time, the discriminator The discriminator evaluates and feeds back the authenticity of the input haze-free image. Evaluate and provide feedback on the authenticity of the input foggy image, the discriminator ( , ) forms a dynamic adversarial relationship with the generator (G, F), driving the generator (G, F) to optimize the conversion strategy to improve the quality and realism of the output image, and finally obtains the AtmoGAN Haze dataset consisting of the target haze image output by the generator G and the target haze-free image output by the generator F;

[0012] S02: Input the constructed AtmoGAN Haze dataset into the target recognition and positioning model to obtain the target recognition and positioning results in foggy weather;

[0013] The target recognition and positioning model includes a model backbone, two model necks and a model detection head; the model backbone is used to convert images in the AtmoGAN Haze dataset into multi-layer feature maps; the first model neck upsamples the high-level semantic features of the feature map layer by layer and fuses them with low-level features through a top-down path and lateral connections; the second model neck transfers the information of low-level features to high-level features through a bottom-up path, which complements the top-down path of the first model neck to achieve bidirectional feature fusion; the model detection head head is used to output a prediction box, which includes a bounding box coordinate, a target category and a confidence score.

[0014] Further: S02 includes:

[0015] After the first convolution layer Conv (Convolution), the image in the AtmoGAN Haze dataset is converted into a feature map of 320×320×64; then after the first hierarchical feature extraction module HFE (Hierarchical Feature Extractor), it is converted into a feature map of 160×160×128; after the second HFE module, it is converted into a feature map of 80×80×256; after the third HFE module, it is converted into a feature map of 40×40×512; after another Conv, it is converted into a feature map of 20×20×1024, and then enters the deformable residual module DRCB (Deformable Residual ConvolutionBlock) module to adjust the receptive field in a targeted manner, reduce information loss, and better extract features; then enters the fast spatial pyramid pooling module SPPF (Fast Spatial Pyramid Pooling Mudule), performs pooling operations of different sizes on the feature map to obtain a set of feature maps of different sizes; then the feature maps are connected together to form a feature map including multi-scale information; and the dimension is reduced through the fully connected layer, and the size of the final feature map remains unchanged;

[0016] The first Conv is converted into a 20×20×512 feature map through the first model neck, and the spatial size of the feature map is expanded to twice the original size, that is, 40×40×512, through upsampling. In the next Cat (Concat), the upsampled feature map is concatenated with the feature map output from the third HFE module in the backbone of the model in the channel dimension to obtain a 40×40×512 feature map; then the first hierarchical feature refinement module (HFR) is used to refine the feature map. Refiner) is converted to a 40×40×256 feature map, and then converted to an 80×80×256 feature map through the second upsample, which is fused with the feature map of the second HFE module in the second Cat to obtain a 80×80×256 feature map; then through the second HFR module, a 80×80×128 feature map is obtained; through the third upsampling, a 160×160×128 feature map is obtained; and it is fused with the feature map output by the first HFE module in the third Cat to obtain a 160×160×128 feature map;

[0017] Entering the second model neck, the 160×160×128 feature map passes through the first C3 (CSPBottleneck With 3 Convolutions) module; then it passes through a Conv to obtain a 80×80×128 feature map, which is Cat-ed with the output of the second HFR module of the first model neck to obtain a 80×80×128 feature map, which passes through the second C3 module to obtain a 80×80×256 feature map; then it passes through the Atrous Convolutional Feature Fusion Module ACFF (Atrous Convolutional Feature Fusion), get a 40×40×256 feature map; Cat with the output of the first HFR module of the neck of the first model to get a 40×40×256 feature map, pass through the third C3 module, get a 40×40×512 feature map; then pass through Conv to get a 20×20×512 feature map; Cat with the output of the first Conv of the neck of the first model to get a 20×20×512 feature map, finally pass through the fourth C3 module to get a 20×20×1024 feature map;

[0018] The model detection head includes four detection heads, which are located behind the four C3 modules of the neck of the second model. The output result of each C3 module corresponds to a detection head, and each detection head independently outputs a set of prediction boxes. Each prediction box includes the bounding box coordinates, target category and confidence score; among them, the detection head after the first C3 module is used to detect extremely small targets, the detection head after the second C3 module is used to detect small targets, the detection head after the third C3 module is used to detect medium targets, and the detection head after the fourth C3 module is used to detect large targets.

[0019] Further: The loss functions included in the migration generation model are:

[0020] Adversarial Loss:

[0021] The adversarial loss corresponding to the mapping of the generator G from the source domain to the target domain for:

[0022]

[0023] The adversarial loss corresponding to the mapping of the generator F from the target domain to the source domain for:

[0024]

[0025] Cycle consistency loss : Cycle consistency loss ensures that the conversion from one domain to another and then back to the original domain can be close to the original image, defined as:

[0026]

[0027] Transmittance uniformity loss : Set the transmittance of the image to a fixed value and compare the target transmittance of the generated image, which is defined as:

[0028]

[0029] is a fixed transmittance, is the target transmittance;

[0030] Loss of consistency in atmospheric light intensity :

[0031]

[0032] is a fixed value of atmospheric light intensity, Target atmospheric light intensity;

[0033] Scattering coefficient guided loss : Set multiple β values ​​to correspond to foggy scenes with different fog levels

[0034]

[0035] Is a function that quantizes the generated image the degree of consistency with the expected scattering effects;

[0036] Atmospheric light scattering loss is defined as :

[0037]

[0038] in, , , is the weight to balance the various losses;

[0039] Loss function for the transfer generative model of atmospheric light scattering priors Converts to:

[0040]

[0041] in:

[0042] : is the adversarial loss of the generator G (mapping from the source domain to the target domain);

[0043] : is the adversarial loss of the generator F (mapping from the target domain to the source domain);

[0044] : is the cycle consistency loss, where is a weight factor used to balance the importance of this term with other loss terms; The reversibility of the mapping in the two directions is measured to ensure that the image can basically maintain its original appearance after being converted between the two domains, thus avoiding meaningless mapping;

[0045] : is the atmospheric light scattering loss, is its weight factor.

[0046] Further: the DRCB module includes:

[0047] 1x1 convolution layer: The input feature map is passed through a 1x1 convolution layer to reduce the number of channels;

[0048] 3x3 standard convolution layer: feature extraction is performed through a 3x3 standard convolution layer;

[0049] Deformable convolutional layer: Dynamically adjust the sampling position through the learned offset, so that the network can adaptively focus on the specific feature direction and geometry of the target;

[0050] Residual connection: add the input features after dimensionality reduction operation to the output of the deformable convolution layer;

[0051] Channel splicing: combine the above processed features with the feature map before dimensionality reduction;

[0052] Final convolutional layer: The output of the entire process is processed through the convolutional layer to generate the final feature map.

[0053] Further: the ACFF module comprises:

[0054] Preliminary feature extraction:

[0055] The input features are divided into two parts; one part directly enters the dilated convolution path, and the other part first enters the maximum pooling path;

[0056] Dilated convolution path:

[0057] In the dilated convolution path, two dilated convolution layers with different dilation rates (1 and 3) are used. The convolution with a dilation rate of 1 acts directly on the input feature map to capture local features and detail information. A maximum pooling layer is added after the two dilated convolution layers. The maximum pooling layer reduces the spatial resolution of the feature map by selecting the maximum value in each pooling window, thereby reducing the amount of calculation while retaining important features.

[0058] Max pooling path:

[0059] Another part of the input features is processed through the maximum pooling operation, which reduces the spatial resolution of the feature map, thereby reducing the amount of calculation while maintaining important structural information;

[0060] Feature Fusion:

[0061] The feature maps after dilated convolution and maximum pooling are merged, and the output of the entire process is processed by a convolution layer to generate the final feature map.

[0062] Compared with the prior art, the present invention has the following technical advances:

[0063] The collection of foggy scene data is a time-consuming and tedious task, and marking targets in foggy data also has a high cost. Therefore, the present invention synthesizes a data set under foggy scenes based on real normal weather data. First, a migration generation model of atmospheric light scattering prior is designed, and fog-free data, transmittance, atmospheric light intensity, and scattering coefficient are used to synthesize foggy data sets of different degrees to increase the richness of the data set. Then, based on this data set sample, a target recognition and positioning model is designed, and the model weights and test sets with the highest accuracy are used to evaluate the model to obtain the foggy target recognition and positioning results. At the same time, on the basis of ensuring the detection speed, the problem of missing data details, reduced contrast, color distortion, etc. in foggy scenes that make it difficult to identify traffic targets is effectively solved, and the accuracy and speed of foggy traffic target recognition and positioning are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0065] In the attached picture:

[0066] Figure 1 is a flow chart of the present invention;

[0067] Figure 2 This is a structural diagram of the atmospheric scattering synthesis fog model of the present invention;

[0068] Figure 3 This is a structural diagram of the target recognition and positioning model of the present invention;

[0069] Figure 4 It is a structural diagram of the deformable residual DRCB module of the present invention;

[0070] Figure 5 This is a structural diagram of the dilated convolution fusion ACFF module of the present invention;

[0071] Figure 6 This is a comparison chart of the detection effects of the present invention. DETAILED DESCRIPTION

[0072] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0073] like Figure 1 As shown, the traffic target recognition and positioning method based on foggy scene includes:

[0074] S01: Use the migration generative model of atmospheric light scattering prior to process the acquired original haze-free images and original haze images to construct the AtmoGAN Haze dataset;

[0075] S02: Input the constructed AtmoGAN Haze dataset into the target recognition and positioning model to obtain the target recognition and positioning results in foggy weather.

[0076] Specifically, the S01 operation steps are as follows:

[0077] First, two sets of image datasets are needed, one set is fog-free natural traffic scene images, that is, original fog-free images, and the other set is images in foggy traffic scenes, that is, original foggy images.

[0078] like Figure 2 As shown in Figure 2, the migration generation model includes an atmospheric scattering model and a generation adversarial network. The generation adversarial network includes two generators (G, F) and two discriminators ( , ).

[0079] Generator G includes an encoder, a residual network, and a decoder, which is responsible for converting the original fog-free image into the target foggy image (i.e., "Generated_x"), while generator F attempts to restore the original foggy image to the target fog-free image (i.e., "Generated_y"). That is, generator G and generator F are responsible for the conversion tasks from source domain to target domain and from target domain to source domain respectively, and the discriminator and the discriminator Judge the authenticity of the generated data and promote the conversion process of the adversarial generative network.

[0080] Specifically, the generator G includes an encoder, a residual network, and a decoder. The encoder extracts features from the original fog-free image, that is, the input original fog-free image is passed through multiple convolutional layers to gradually reduce its spatial resolution while increasing the number of feature channels to capture high-level features of the image (such as shape, contour, and semantic information). It is then converted into a feature map through a residual network, that is, using a residual block structure (Resnet blocks) to learn the mapping from the fog-free image domain to the foggy image domain while maintaining the basic structure of the image content.

[0081] In order to effectively integrate the atmospheric parameters into the generation process, the atmospheric parameters are encoded into a vector [c=t, A, β] (transmittance t, atmospheric light value A and scattering coefficient β) that matches the dimension of the internal feature map of the generator network, and the encoded atmospheric parameters are fused with the feature map after the residual block structure. That is, the transmittance t and the atmospheric light value A are set, and different scattering coefficient β values ​​are selected according to the comparison of actual foggy scene samples, and the fog concentration is adjusted to achieve precise control of the fog effect. In this way, the atmospheric parameters can be used to guide the simulation of the fog effect, and the fog effect can be added or reduced as needed when generating the image.

[0082] The specific operation is to add the atmospheric parameters as weight factors after the deconvolution layer and multiply them with each element of the feature map to adjust the intensity or characteristics of the feature map and further fine-tune the image's color, brightness, contrast and other attributes to ensure that the final output image not only has realistic fog effects, but is also highly consistent with the desired fog effects.

[0083] The generation process of generator F is the same as that of generator G, and also includes an encoder, a residual network, and a decoder, that is, multiple convolutional layers, residual blocks, and deconvolutional layers, which are used to learn the mapping from the original foggy image to the target fog-free image without adding additional atmospheric parameter vectors.

[0084] In order to overcome the high uncertainty of the mapping and avoid mode collapse, a cycle consistency loss is introduced. This strategy requires that the transformed image can be close to the original state after inverse transformation (i.e., F(G(X))≈X and G(F(Y))≈Y) to constrain and guide model learning.

[0085] The role of cycle consistency loss is not limited to maintaining the coherence of the transformation. It also promotes the model's retention of image content and precise control of style transfer. Through this mechanism, the model is forced to deeply understand the intrinsic relationship between the two image domains, thereby improving its generalization ability on unseen images while suppressing the tendency to overfitting. In addition, the inverse transformation process, as part of two-way learning, provides the model with additional learning signals to help it more comprehensively capture and imitate cross-domain image features, ensuring that the core content and structure of the image are protected when adding or removing specific visual effects (such as fog). In practice, cycle consistency loss reduces the generation of unnecessary content and promotes the improvement of image quality by ensuring that the semantic information remains unchanged during the transformation process, making the generated images more visually realistic and rich in details. This mechanism not only strengthens the accuracy of the model's transformation between the two domains, but also significantly enhances its ability to process new data by increasing the model's understanding of shared features.

[0086] The main task of the discriminator is to distinguish whether the data input to it comes from the real data distribution or the fake data generated by the generator. It tries to learn the characteristics of the real data so that it can accurately identify which data is real and which is forged by the generator. In the discriminator network, the first four convolutional layers are used to extract the output features of the generative model, and then a convolutional layer that produces a one-dimensional output is added to determine whether the generated data conforms to the data distribution of the real sample.

[0087] Discriminator The discriminator evaluates and feeds back the authenticity of the input haze-free image. Evaluate and provide feedback on the realism of the input foggy image. During the training process, the discriminator indirectly helps the generator improve its performance by providing feedback to it. When the discriminator can easily distinguish between real data and fake data, it sends a signal to the generator that "the data is not real enough". This feedback prompts the generator to adjust its parameters to generate more realistic fake data that is difficult for the discriminator to identify.

[0088] In this application, a competitive relationship is formed between the discriminator and the generator. The generator strives to generate more and more realistic data to deceive the discriminator, while the discriminator strives to improve its discrimination ability to distinguish between true and false data. This competitive relationship promotes the continuous optimization and improvement of the model. The loss generated by the interaction between the discriminator and the generator is the adversarial loss. The adversarial loss helps the model learn the image distribution in the target field.

[0089] The discriminator indirectly promotes the diversity of generated data by encouraging the generator to generate diverse fake data. If the generator always generates similar types of fake data, the discriminator will be able to easily identify them, forcing the generator to try to generate more diverse data to deceive the discriminator. During the training process, the generator and the discriminator are optimized alternately. The generator tries to generate images that can deceive the discriminator, while the discriminator tries to more accurately identify which images are real and which are generated by the generator.

[0090] The final output is the target foggy image output by generator G and the target fog-free image output by generator F. The target foggy image is visually similar to the original foggy image in the target domain, but is actually converted from the original fog-free image in the source domain. The target foggy image and the target fog-free image together constitute the AtmoGAN Haze dataset.

[0091] The specific synthesis process of the atmospheric scattering model includes the following steps: First, the transmittance t and the atmospheric light intensity A are set. Starting from the known fog-free data, the transmittance and the atmospheric light intensity A are set, and the influence of fog is introduced by simulating the scattering effect of particles in the atmosphere. Secondly, in order to increase the fidelity of the synthetic data set, different scattering coefficient β values ​​are selected according to the comparison of actual foggy scene samples, and the concentration of fog is adjusted. In this embodiment, multiple β values ​​are set, corresponding to foggy scenes with different foggy degrees, such as β of 0.025, 0.05, 0.075, 0.100, 0.125 and 0.150. Then, in order to improve the authenticity of the generated foggy data, the global atmospheric light is adjusted to simulate the scattering and reflection effects of light in foggy days, so that the synthesized foggy data is more realistic. Finally, the center of the fog is set at the midpoint coordinate of the upper boundary of the foggy data to ensure that the distribution and aggregation of the fog are closer to the actual scene.

[0092] In foggy scenes, this application focuses on whether the generated data is realistic, especially the characteristics in foggy scenes, while the cycle consistency loss ensures that the cyclically transformed data can maintain the original structure and content. In foggy data generation, it helps to ensure that the generated results maintain authenticity in the transition between defogging and fogging.

[0093] The migration generation model of atmospheric light scattering prior includes atmospheric light scattering loss, adversarial loss, cycle consistency loss, etc. The atmospheric light scattering loss ensures that the generated foggy image is physically reasonable by calculating the difference between the transmittance of the generated image and the transmittance of the actual foggy image, as well as the consistency of the atmospheric light intensity. During training, the scattering coefficient is randomly changed to simulate different foggy conditions. When the generator and discriminator are optimized to update the parameters, the degree of fogging of the generated image is adjusted according to the atmospheric parameters to ensure accurate simulation and restoration under different foggy conditions.

[0094] The adversarial loss corresponding to the mapping of the generator G from the source domain to the target domain for:

[0095]

[0096] The generator G tries to generate images similar to those in the target domain, while the discriminator strives to distinguish between real images y and images generated by G. .

[0097] The adversarial loss corresponding to the mapping of the generator F from the target domain to the source domain for:

[0098]

[0099] Here, the generator F generates images similar to the source domain, while the discriminator distinguishes between real images x and images generated by F. .

[0100] Cycle consistency loss ensures that the conversion from one domain to another and then back to the original domain can be close to the original image. Defined as:

[0101]

[0102] This loss encourages is close to x, and Close to y, that is, the forward and backward cycle consistency is completed

[0103] Transmittance uniformity loss : Set the transmittance of the image to a fixed value and compare the target transmittance of the generated image, which is defined as:

[0104]

[0105] is a fixed transmittance, is the target transmittance.

[0106] Loss of consistency in atmospheric light intensity :

[0107]

[0108] is a fixed value of atmospheric light intensity, Target atmospheric light intensity.

[0109] Scattering coefficient guided loss :

[0110] Set multiple β values ​​to correspond to foggy scenes with different fog levels, such as β of 0.025, 0.05, 0.075, 0.100, 0.125, and 0.150

[0111]

[0112] Is a function that quantizes the generated image Quantifies how well the generated image matches the expected scattering effects.

[0113] In summary, atmospheric light scattering loss It can be defined as:

[0114]

[0115] in, , , It is the weight to balance the various losses.

[0116] Transfer generative model of atmospheric light scattering prior, overall loss function Converts to:

[0117]

[0118] in:

[0119] : is the total loss function, which combines all sub-loss terms and aims to optimize the entire model for the unpaired image-to-image translation task.

[0120] : is the adversarial loss of the generator G (the mapping from the source domain to the target domain), which is related to the discriminator. Its goal is to make the images produced by G look as much like real images from the target domain as possible, making it difficult to distinguish them.

[0121] : is the adversarial loss of the generator F (mapping from the target domain to the source domain), which is related to the discriminator. Its function is similar to, but in the opposite direction, the purpose is to make the images generated by F difficult to distinguish, as if they are real samples of the X domain.

[0122] : is the cycle consistency loss, where is a weight factor used to balance the importance of this term with other loss terms, measuring whether the mapping in the two directions is reversible to ensure that the image can basically maintain its original appearance after being converted between the two domains, thereby avoiding meaningless mapping.

[0123] : is the atmospheric light scattering loss, and is its weight factor, which is used to ensure that the generated image can still maintain physical rationality after considering the atmospheric scattering effect under foggy conditions.

[0124] Specifically, the S02 operation steps are as follows:

[0125] The target recognition and positioning model structure is as follows: Figure 3As shown, there are 4 columns in total. The first column is the model backbone, the second and third columns are the model neck, and the fourth column is the model detection head. The image size of the input model will be converted to 640×640 (image width and height), and they are all visible light images with 3 channels, so the input image parameters are 640×640×3. The model backbone is used to convert images in the AtmoGAN Haze dataset into multi-layer feature maps; the first model neck upsamples the high-level semantic features of the feature map layer by layer and fuses them with low-level features through a top-down path and lateral connections; the second model neck transfers the information of low-level features to high-level features through a bottom-up path, which complements the top-down path of the first model neck to achieve bidirectional feature fusion; the model detection head is used to output the prediction box, which includes the bounding box coordinates, target category and confidence score

[0126] Specifically, the image first enters the backbone of the model, whose main function is to convert the input image into a multi-layer feature map and extract feature information. After the first Conv, the image becomes a 320×320×64 feature map. The output of the number of channels is determined by the parameters of the model. The convolution layer is used to extract local information from the input features. Then it passes through the first HFE module and becomes a 160×160×128 feature map. The HFE module improves the feature extraction capability by increasing the depth and receptive field of the network, and performs feature extraction at the same time. After the second HFE module, it becomes an 80×80×256 feature map, and after the third HFE module, it becomes a 40×40×512 feature map. After another Conv, it becomes a 20×20×1024 feature map, and then enters the DRCB module to adjust the receptive field in a targeted manner, reduce information loss, and better extract features. Then it enters the SPPF module and performs pooling operations of different sizes on the feature maps to obtain a set of feature maps of different sizes. Finally, these feature maps are connected together to form a feature map containing multi-scale information, and the dimension is reduced through the fully connected layer. The size of the final feature map remains unchanged.

[0127] Backbone performs dimensionality reduction through Conv and HFE modules, extracting image features during the dimensionality reduction process. This is done to:

[0128] ① Reduce the amount of calculation: When the image passes through the convolution layer, the dimension will gradually decrease. Through dimensionality reduction, the spatial resolution of the feature map is reduced, thereby reducing the amount of calculation and memory usage of subsequent layers.

[0129] ② Extract high-level features: Dimensionality reduction helps to extract high-level features of the image. Smaller feature maps contain higher-level information, while larger feature maps usually contain more detail information. By gradually reducing the dimensionality, unnecessary detail noise can be suppressed while retaining key features.

[0130] ③ Improve the generalization ability of the model: Appropriate dimensionality reduction helps the model better capture the core features of the data without overfitting the training data, which is crucial for the model to better generalize when processing unseen data.

[0131] ④ Merge multi-scale features: Through dimensionality reduction, features of different scales can be generated, providing conditions for the fusion of features of different scales in the subsequent neck layer.

[0132] Then the feature map comes to the second column, which is the neck of the first model. It is transformed into a 20×20×512 feature map through the first Conv in the second column. The spatial size of the feature map is expanded to twice the original size, that is, 40×40×512, through upsampling. At the next Cat, the upsampled feature map is spliced ​​with the feature map output from the third HFE module in the backbone in the channel dimension to obtain a 40×40×512 feature map. This is done to fuse the deeper feature map with the shallower feature map, thereby combining high-level semantic information and low-level detail information. Then through the first HFR module, the HFR module improves the feature extraction capability by increasing the depth and receptive field of the network, and performs feature extraction at the same time. It becomes a 40×40×256 feature map, and then through the second upsample, it becomes a 80×80×256 feature map, which is fused with the feature map of the second HFE module in the second Cat to obtain a 80×80×256 feature map. Then through the second HFR module, a 80×80×128 feature map is obtained. Through the third upsampling, a feature map of 160×160×128 is obtained, which is fused with the feature map output by the first HFE module in the third Cat to obtain a feature map of 160×160×128.

[0133] The second column upsamples the high-level semantic features layer by layer and fuses them with the low-level features through a top-down path and lateral connections, allowing the model to simultaneously utilize high-resolution detail information and low-resolution semantic information, which helps to improve the detection accuracy of objects of different sizes.

[0134] Entering the third column, which is the second model neck, the 160×160×128 feature map passes through the first C3 module. The main function of the C3 module is to increase the depth and receptive field of the network and improve the ability of feature extraction. It is still 160×160×128. Then, after a Conv, a 80×80×128 feature map is obtained. It is Cat with the output of the second HFR module in the second column to obtain a 80×80×128 feature map. After the second C3 module, a 80×80×256 feature map is obtained. After passing through the ACFF module, a 40×40×256 feature map is obtained. The ACFF module expands the receptive field to better retain the local detail features of the target. Cat with the output of the first HFR module in the second column to obtain a 40×40×256 feature map. After the third C3 module, a 40×40×512 feature map is obtained. After Conv, a 20×20×512 feature map is obtained. Cat is performed with the output of the first Conv in the second column to obtain a feature map of 20×20×512, and finally passes through the fourth C3 module to obtain a feature map of 20×20×1024.

[0135] The third column is used to enhance multi-scale feature fusion. Its main function is to transfer the information of low-level features to high-level features through a bottom-up path, which complements the top-down path of the second column to achieve bidirectional feature fusion. The multi-scale feature fusion capability of the second column significantly enhances the detection effect of small targets, because low-level features have higher resolution and detail information, which are transferred to high-level features through the third column, helping to improve the detection accuracy of small targets. In addition, through the bidirectional fusion of multi-scale features, the third column provides a richer feature representation, thereby improving the overall detection accuracy of the model.

[0136] The fourth column is the model detection head. There are four detection heads, which are located behind the four C3 modules in the third column. The output of each C3 module corresponds to a detection head. Each detection head will independently output a set of prediction boxes, each prediction box contains the bounding box coordinates, target category and confidence score. Among them, the detection head after the first C3 module is used to detect extremely small targets, the detection head after the second C3 module is used to detect small targets, the detection head after the third C3 module is used to detect medium targets, and the detection head after the fourth C3 module is used to detect large targets.

[0137] DRCB module such as Figure 4 As shown, the specific process includes:

[0138] 1x1 convolution layer: First, the input feature map passes through a 1x1 convolution layer Conv to reduce the dimension of the number of channels. This design helps to reduce the number of model parameters and reduce computational complexity while maintaining the transmission of key information.

[0139] 3x3 standard convolution layer: Next, feature extraction is performed through a 3x3 standard convolution layer. This operation helps reduce computational complexity while maintaining feature richness.

[0140] Deformable convolution layer: This is followed by the Deformable Convolution layer DConv (Deformable Convolution), which dynamically adjusts the sampling position through the learned offset, so that the network can adaptively focus on the specific feature direction and geometry of the target. Even when the target is deformed or scaled, it can more accurately capture details and structural information, greatly enhancing the adaptability and expressiveness of the model. Specifically, it includes:

[0141] (1) Offset learning, deformable convolution is achieved by combining with input features Figure 1 The training starts with learning the offset at each position. These offsets are used to adjust the sampling point position of the convolution kernel. (2) Sampling point offset: For each position, a new sampling point position can be obtained by interpolating and accumulating the learned offsets. Since the sampling point position is dynamically adjusted according to the learned offsets, the convolution kernel can adapt to the irregular shape and spatial changes of the target at different positions.

[0142] Each position on the standard convolution output feature map , calculated by the following first formula, in deformable convolution, given the input feature map, a temporary value is obtained by bypass convolution, and each on the deformable convolution The second formula is used to calculate the sampling position. After calculation, the sampling position is converted to an irregular position. Due to the offset It is usually a very small number, so it is implemented by bilinear interpolation, as shown in the fourth formula below:

[0143]

[0144]

[0145]

[0146] Among them, the grid R represents the size of the receiving field, represents the position in R, is any position on the input feature map, is the weight of the convolution kernel, is the offset, q represents all the integral spatial positions in the feature map x, and G is the bilinear interpolation.

[0147] Residual connection: In order to retain the position information and avoid information loss, the module adopts the idea of ​​residual network, adding the input features after dimensionality reduction operation to the output of the deformable convolution layer, and Add is the addition of feature maps. This not only retains the original position information, but also avoids information loss through the residual learning strategy, and strengthens the ability to capture target details.

[0148] Channel splicing: In order to achieve multi-level fusion and complementarity of features, the module combines the above processed features with the feature map before dimensionality reduction through channel splicing Cat operation. This not only enriches the feature expression of the model, but also promotes the complementarity of information at different scales and levels, further improving the recognition and segmentation accuracy of the model in complex scenarios.

[0149] Final convolutional layer: Finally, the output of the entire process is processed through a convolutional layer to generate the final feature map.

[0150] Specifically, the ACFF module is as follows Figure 5 Shown include:

[0151] The dilated convolution can effectively avoid the reduction of resolution. By adjusting the dilated convolution parameters, the size of the receptive field can be flexibly changed to better capture contextual information, thereby more effectively extracting important features, enhancing the model's recognition and positioning effect for small targets in complex scenes, and improving the model's generalization ability. In addition, the maximum pooling operation is also an important feature processing method, which can retain local detail features and avoid feature information offset caused by convolution operations. In summary, the dilated convolution fusion module ACFF module of the present application, such as Figure 4 As shown in the figure, this module combines the advantages of dilated convolution and maximum pooling to effectively avoid grid effects, so as to better process feature maps and improve model performance. The ACFF module enables the model to effectively process target context information while maintaining high-resolution feature maps, thus improving the model's ability to process feature detail information. The calculation formulas for standard convolution and dilated convolution are as follows:

[0152]

[0153]

[0154] Where f is the kernel size, which is a discrete function of m, g is a filter of size n, r represents the size of the receptive field, and d represents the dilation rate.

[0155] The specific processing flow of the ACFF module includes:

[0156] Preliminary feature extraction:

[0157] The input features are divided into two parts, one part directly enters the AtrousConvolution (AConv) path, and the other part first enters the Max Pooling (Max Pooling) path.

[0158] Dilated convolution path:

[0159] In the dilated convolution path, two dilated convolution layers with different dilation rates (1 and 3) are used. Convolution with a dilation rate of 1 means that no additional holes are inserted, so it behaves like a standard convolution. It acts directly on the input feature map to capture local features and detail information. Convolution with a dilation rate of 3 means that the spacing between filters is larger, which can cover a wider area on the input feature map, effectively increasing the receptive field. This design enables the layer to capture more contextual information and helps to identify larger-scale features and patterns, which is very important for understanding the global structure in the feature map while avoiding the "grid effect". Two dilated convolution layers are usually followed by a maximum pooling layer. Maximum pooling reduces the spatial resolution of the feature map by selecting the maximum value in each pooling window, thereby reducing the amount of computation, while retaining important features (such as the maximum intensity of edges or textures), which helps prevent overfitting, increase the generalization ability of the model, and can maintain attention to significant features to a certain extent.

[0160] Max pooling path:

[0161] Another part of the input features is processed through the maximum pooling operation to reduce the spatial resolution of the feature map, thereby reducing the amount of computation while maintaining important structural information.

[0162] Feature Fusion:

[0163] The feature maps after dilated convolution and maximum pooling are merged, which occurs in the Cat module. Feature fusion allows the model to simultaneously utilize features from high resolution and large contexts, improving the ability to recognize multi-scale objects.

[0164] Finally, the output of the entire process is processed through a convolutional layer Conv to generate the final feature map.

[0165] The verification process is as follows:

[0166] The data set synthesized in S01 is divided into a training set and a test set at a ratio of 8:2, and input into the ADDS-Net target detection model for training. The number of training iterations is set to 200, and the optimal training weight is saved during the training process, and then used as the input weight of the test, and then input into the test set to obtain the prediction result.

[0167] The accuracy evaluation indicators of the model experimental detection are measured by precision P (Precision), recall R (Recall), average precision AP (Average Precision), mean average precision mAP (mean Average Precision), parameters (params), and frames per second FPS. The calculation methods of P, R, AP and mAP are as follows:

[0168]

[0169]

[0170]

[0171]

[0172] Among them: TP is true positive, FP is false positive, FN is false negative, AP is the average precision of a single category, and mAP is the average AP of all categories.

[0173] In order to verify the effectiveness of the proposed module, ablation experiments were carried out on the dilated convolution feature fusion module and the deformable residual module, and it can be seen that the model accuracy has been improved.

[0174] Table 1 Ablation experiment

[0175] Model mAP people Rider car the bus truck bike motorcycle params ADDS-Net 66.4% 49.6% 69.9% 80.7% 66.7% 60.1% 63.5% 74.3% 58.7M -DRCB 66.1% 48.5% 71.0% 79.9% 63.0% 60.0% 64.9% 75.3% 53.5M -ACFF 66.1% 49.5% 69.0% 80.0% 65.4% 64.2% 65.5% 68.9% 57.9M

[0176] It can be clearly seen from Table 1 that the two modules proposed in the present invention improve the detection accuracy of the algorithm model. In order to verify the effectiveness of the DRCB module, it was removed, and the results showed that the mAP decreased by 0.3% and the params decreased by 5.2M. It is particularly noteworthy that the decrease in targets such as buses is large, which further proves that the DRCB module has a significant effect on the recognition and positioning of these targets. In order to verify the effectiveness of the ACFF module, this module was also removed. The results showed that the overall mAP of the model decreased by 0.4%, among which small targets such as motorcycles decreased more significantly. The ablation experiment results further confirmed that the proposed module improves the detection accuracy without adding too many parameters. The detection effect is compared, as shown in the figure. Figure 6 shown.

[0177] In order to verify the effectiveness of the detection model, ADDS-Net is compared with some mainstream deep learning networks, including Faster R-CNN (Faster Region-based Convolutional Neural Networks), SSD (Single Shot MultiBox Detector), YOLOV3, YOLOV3-SPP (Spatial Pyramid Pooling, SPP), YOLOV4, YOLOV7, YOLOV8 and DETR (Detection Transformer). The experimental results are shown in Table 2. It can be clearly seen that the ADDS-Net algorithm proposed in the present invention has a much higher mAP than other models when compared with other mainstream detection models, almost twice that of SSD. Although Faster R-CNN represents a two-stage detection algorithm with good detection accuracy, its detection accuracy is far less than that of ADDS-Net and is not suitable for real-time applications. In contrast, although YOLOV3, YOLOV3-SPP and YOLOV4 have faster detection speeds, their detection accuracy is low and the overall effect is not good. Compared with YOLOV5X, ADDS-Net performs well in detection accuracy and recall rate, and the number of parameters is much smaller than YOLOV5X. Compared with YOLOV7, YOLOV8 and DETR, although the number of parameters is not dominant, it shows the best performance in detection accuracy, recall rate and detection speed. In summary, the ADDS-Net algorithm proposed in this invention has achieved the best level in detection accuracy. Although the detection speed is slightly lower than that of some YOLO series models, it is still the best choice in terms of comprehensive performance.

[0178] Table 2 Comparative experiment

[0179] Model P R mAP params FPS Faster R-CNN 56.7% 41.3% 56.3% 60.2M 16.4 SSD 34.9% 28.5% 34.9% 68.0M 30.3 YOLOV3 75.6% 54.6% 61.2% 61.5M 48.0 YOLOV3_SPP 76.4% 53.9% 59.4% 62.6M 45.5 YOLOV4 70.6% 44.0% 50.8% 64.4M 50.1 YOLO V5X 71.0% 44.3% 51.0% 86.5M 26.7 YOLO V7 74.0% 58.1% 62.8% 36.5M 42.5 YOLOV8 70.2% 53.3% 59.5% 43.6M 35.3 DETR 63.3% 44.2% 46.7% 32.0M 28.4 YOLO-Z 71.3% 46.8% 52.9% 55.6M 28.8 ours 78.4% 53.7% 66.4% 54.2M 35.6

[0180] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the claims of the present invention.

Claims

1. A traffic target recognition and positioning method based on foggy scenes, characterized in that: include: S01: Use the migration generative model of atmospheric light scattering prior to process the acquired original haze-free images and original haze images to construct the AtmoGAN Haze dataset; The migration generation model includes an atmospheric scattering model and an adversarial generation network, wherein the adversarial generation network includes two generators (G, F) and two discriminators ( , ); The generator G includes an encoder, a residual network and a decoder. The encoder extracts features from the original fog-free image and then converts it into a feature map through the residual network. The feature map is deconvolved by the decoder and fused with the atmospheric parameters generated by the atmospheric scattering model. After convolution, the target foggy image is output, and a cycle consistency loss is introduced so that the content of the output target foggy image is consistent with the input image. The generator F includes an encoder, a residual network and a decoder. The encoder extracts features from the original foggy image and converts it into a feature map through the residual network. The feature map is deconvolved through the decoder and then outputs a target fog-free image after convolution. Cycle consistency loss is introduced so that the content of the output target fog-free image is consistent with the input image. At the same time, the discriminator The discriminator evaluates and feeds back the authenticity of the input haze-free image. Evaluate and provide feedback on the authenticity of the input foggy image, the discriminator ( , ) forms a dynamic adversarial relationship with the generator (G, F), driving the generator (G, F) to optimize the conversion strategy to improve the quality and realism of the output image, and finally obtains the AtmoGAN Haze dataset consisting of the target haze image output by the generator G and the target haze-free image output by the generator F; S02: Input the constructed AtmoGAN Haze dataset into the target recognition and positioning model to obtain the target recognition and positioning results in foggy weather; The target recognition and positioning model includes a model backbone, two model necks, and a model detection head; the model backbone is used to convert images in the AtmoGAN Haze data set into multi-layer feature maps; the first model neck upsamples the high-level semantic features of the feature map layer by layer and fuses them with low-level features through a top-down path and lateral connections; the second model neck transmits the information of the low-level features to the high-level features through a bottom-up path, which complements the top-down path of the first model neck to achieve bidirectional feature fusion; The model detection head is used to output a prediction box, which includes a bounding box coordinate, an object category, and a confidence score.

2. The method for identifying and locating traffic targets based on foggy scenes according to claim 1 is characterized in that: The S02 includes: After the first Conv, the images in the AtmoGAN Haze dataset are converted into a 320×320×64 feature map; then after the first HFE module, they are converted into a 160×160×128 feature map; after the second HFE module, they are converted into an 80×80×256 feature map; after the third HFE module, they are converted into a 40×40×512 feature map; after another Conv, they are converted into a 20×20×1024 feature map, and then enter the DRCB module to adjust the receptive field in a targeted manner, reduce information loss, and better extract features; then enter the SPPF module to perform pooling operations of different sizes on the feature map to obtain a set of feature maps of different sizes; then the feature maps are connected together to form a feature map that includes multi-scale information; and the dimension is reduced through the fully connected layer, and the size of the final feature map remains unchanged; The first Conv of the first model neck is converted into a feature map of 20×20×512, and the spatial size of the feature map is expanded to twice the original size, that is, 40×40×512, through upsample. At the next Cat, the upsampled feature map is spliced ​​with the feature map output from the third HFE module in the backbone of the model in the channel dimension to obtain a 40×40×512 feature map; then it is converted into a 40×40×256 feature map through the first HFR module, and then converted into a 80×80×256 feature map through the second upsample, and fused with the feature map of the second HFE module in the second Cat to obtain a 80×80×256 feature map; then through the second HFR module, a 80×80×128 feature map is obtained; through the third upsampling, a 160×160×128 feature map is obtained; and it is fused with the feature map output from the first HFE module in the third Cat to obtain a 160×160×128 feature map; Entering the second model neck, the 160×160×128 feature map passes through the first C3 module; then it passes through a Conv to obtain a 80×80×128 feature map, which is Cat-ed with the output of the second HFR module of the first model neck to obtain a 80×80×128 feature map, which is passed through the second C3 module to obtain a 80×80×256 feature map; then it passes through the ACFF module to obtain a 40×40×256 feature map; it is Cat-ed with the output of the first HFR module of the first model neck to obtain a 40×40×256 feature map, which is passed through the third C3 module to obtain a 40×40×512 feature map; then it passes through Conv to obtain a 20×20×512 feature map; it is Cat-ed with the output of the first Conv of the first model neck to obtain a 20×20×512 feature map, and finally it passes through the fourth C3 module to obtain a 20×20×1024 feature map; The model detection head includes four detection heads, which are located behind the four C3 modules of the neck of the second model. The output result of each C3 module corresponds to a detection head, and each detection head independently outputs a set of prediction boxes. Each prediction box includes the bounding box coordinates, target category and confidence score; among them, the detection head after the first C3 module is used to detect extremely small targets, the detection head after the second C3 module is used to detect small targets, the detection head after the third C3 module is used to detect medium targets, and the detection head after the fourth C3 module is used to detect large targets.

3. The method for identifying and locating traffic targets based on foggy scenes according to claim 1 is characterized in that: The loss functions included in the migration generation model are: Adversarial Loss: The adversarial loss corresponding to the mapping of the generator G from the source domain to the target domain for: ; The adversarial loss corresponding to the mapping of the generator F from the target domain to the source domain for: ; Cycle consistency loss : Cycle consistency loss ensures that the conversion from one domain to another and then back to the original domain can be close to the original image, defined as: ; Transmittance uniformity loss : Set the transmittance of the image to a fixed value and compare the target transmittance of the generated image, which is defined as: ; is a fixed transmittance, is the target transmittance; Loss of consistency in atmospheric light intensity : ; is a fixed value of atmospheric light intensity, Target atmospheric light intensity; Scattering coefficient guided losses : Set multiple Values, corresponding to foggy scenes with different fog levels ; Is a function that quantizes the generated image the degree of consistency with the expected scattering effects; Atmospheric light scattering loss is defined as : ; in, , , is the weight to balance the various losses; Loss function for the transfer generative model of atmospheric light scattering priors Translates to: ; in: : is the adversarial loss of the generator G (mapping from the source domain to the target domain); : is the adversarial loss of the generator F (mapping from the target domain to the source domain); : is the cycle consistency loss, where is a weight factor used to balance the importance of this term with other loss terms; The reversibility of the mapping in the two directions is measured to ensure that the image can basically maintain its original appearance after being converted between the two domains, thus avoiding meaningless mapping; : is the atmospheric light scattering loss, is its weight factor.

4. The method for identifying and locating traffic targets based on foggy scenes according to claim 2 is characterized in that: The DRCB module includes: 1x1 convolution layer: The input feature map is passed through a 1x1 convolution layer to reduce the number of channels; 3x3 standard convolution layer: feature extraction is performed through a 3x3 standard convolution layer; Deformable convolutional layer: Dynamically adjust the sampling position through the learned offset, so that the network can adaptively focus on the specific feature direction and geometry of the target; Residual connection: add the input features after dimensionality reduction operation to the output of the deformable convolution layer; Channel splicing: combine the above processed features with the feature map before dimensionality reduction; Final convolutional layer: The output of the entire process is processed through the convolutional layer to generate the final feature map.

5. The method for identifying and locating traffic targets based on foggy scenes according to claim 2 is characterized in that: The ACFF module includes: Preliminary feature extraction: The input features are divided into two parts; one part directly enters the dilated convolution path, and the other part first enters the maximum pooling path; Dilated convolution path: In the dilated convolution path, two dilated convolution layers with different dilation rates (1 and 3) are used. The convolution with a dilation rate of 1 acts directly on the input feature map to capture local features and detail information. A maximum pooling layer is added after the two dilated convolution layers. The maximum pooling layer reduces the spatial resolution of the feature map by selecting the maximum value in each pooling window, thereby reducing the amount of calculation while retaining important features. Max pooling path: Another part of the input features is processed through the maximum pooling operation, which reduces the spatial resolution of the feature map, thereby reducing the amount of calculation while maintaining important structural information; Feature Fusion: The feature maps after dilated convolution and maximum pooling are merged, and the output of the entire process is processed by a convolution layer to generate the final feature map.

Citation Information

Cited By

  • Cross-modal interaction conversion method and device for haze remote sensing image target detection

    CN121033643A

  • Target detection method and device based on domain self-adaption, equipment and medium

    CN121708557A