Target detection method for unmanned aerial vehicle in severe weather based on image depth information
By using deep learning style transfer technology and GAN models to enhance image data in severe weather conditions, combined with the feature pyramid and detection head network of deep convolutional neural networks, the problem of low accuracy of drone target detection in severe weather conditions is solved, and efficient target recognition in complex environments is achieved.
Patent Information
- Application Number
- CN202510634225.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-09-19
AI Technical Summary
Under severe weather conditions, the image quality problems obtained by drone image acquisition equipment make it difficult for traditional deep neural networks to accurately identify targets. Existing image processing algorithms cannot effectively cope with the complex noise and image degradation in severe weather conditions, resulting in low target detection accuracy.
Using deep learning-based style transfer technology, the style features of real harsh scenes are transferred to high-quality images taken in clear weather. A GAN model is constructed for image data enhancement, and a deep convolutional neural network is used to extract feature pyramids and detection head networks for target detection, enhancing the generalization ability and robustness of the GAN model.
It effectively solves the problem of poor target detection in bad weather, improves the target detection accuracy and robustness of UAVs in bad weather conditions, and can accurately identify multi-scale targets in complex environments.
Smart Images
Figure CN120673285A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for detecting unmanned aerial vehicle (UAV) targets in severe weather based on image depth information. Background Art
[0002] In recent years, with the advancement of image processing technology, real-time target detection in captured images has become increasingly important. Unmanned aerial vehicles (UAVs) equipped with image acquisition equipment, thanks to their flexible maneuverability and rapid deployment capabilities, have been widely used in a variety of fields, including security patrols, emergency rescue, and geographic mapping. However, the varying target types in these fields not only affect the accuracy of image target detection, but also interfere with target detection when capturing images under varying climatic conditions. This is particularly true when capturing images of targets outdoors, where image quality is inevitably affected by the environment. For example, in inclement weather such as rain, snow, fog, and haze, image acquisition equipment mounted on UAVs is highly susceptible to these conditions, resulting in images often suffering from image quality issues such as high noise, low contrast, and blurred target features. To address this issue, a conventional approach is to preprocess the target image before detecting it. Otherwise, accurate target analysis during image processing becomes difficult in the presence of image quality issues.
[0003] With the rapid development of computer vision technology, deep neural networks have brought revolutionary breakthroughs in object recognition in captured images. While traditional deep neural networks can accurately identify various targets in images captured under clear weather conditions, images captured under inclement weather suffer from severe light scattering and refraction, which distorts the appearance of targets. This in turn renders the recognition algorithms based on single visual features in traditional deep neural networks ineffective. For example, fog blurs the outlines of objects in captured images, making it difficult for computer vision technology to accurately extract effective features related to the targets in the images and to understand the semantic information in the images, making it impossible to reliably distinguish between different targets in the images.
[0004] In addition, although there are many mature algorithms in image processing technology that can be applied to image optimization, most of these simple dehazing algorithms may cause image distortion due to incomplete dehazing or excessive dehazing. Most of the denoising and enhancement algorithms are also difficult to cope with the complex noise models and image degradation patterns unique to severe weather conditions, making it easy for the denoising process of image processing to lose target detail information, and thus unable to provide an ideal image basis for target detection based on depth information. Summary of the Invention
[0005] The present invention aims to address the shortcomings of existing technologies in terms of low accuracy in inclement weather target detection. It provides a method for detecting drone targets in inclement weather based on image depth information. This method employs deep learning-based style transfer technology to transfer the stylistic features of real inclement scenes to high-quality images captured in clear weather. The image data is geometrically transformed, and a GAN model specifically designed for enhancing inclement weather image data is constructed. This method not only expands the diversity of image data but also enhances the generalization and robustness of the GAN model. By using a detection head network to obtain target detection information from images, the present invention effectively addresses the problem of poor target detection results resulting from directly detecting raw images captured in inclement weather.
[0006] To achieve the above objectives, the technical solutions provided by the present invention are:
[0007] A method for detecting UAV targets in severe weather based on image depth information, comprising:
[0008] Step 1: Input the severe image data collected under severe weather conditions into a trained deep convolutional neural network to extract features, and construct a feature pyramid based on the features; upsample and fuse the features of different layers in the feature pyramid to obtain a feature pyramid with multi-scale information;
[0009] Step 2: obtaining a feature map based on the feature pyramid, inputting the feature map into a detection head network, and outputting target detection information corresponding to the bad image data after performing regression analysis and positioning analysis on the feature map;
[0010] Specifically, the training process of the deep convolutional neural network in step 1 includes:
[0011] Step (11), establishing a training sample data set containing severe weather data; wherein the severe weather data in the training sample data set is obtained by the following process: step a, using style transfer technology to perform style transfer on clear image data collected under existing clear weather conditions to obtain an initial severe weather data set; step b, using a deep learning model to enhance the severe weather data set, and using simulation technology to generate a variety of severe weather conditions to obtain a primary expansion data set; performing small target data enhancement on the primary expansion data set, and using geometric transformation technology to perform a more diverse simulation on it to obtain a secondary expansion data set;
[0012] Step (12), building a deep convolutional neural network; specifically, building an input layer, a convolutional layer, a feature extraction layer and a detection head layer in sequence, the built input layer receives the bad weather image data obtained through style transfer, GAN model enhancement and small target data enhancement operations; the built convolutional layer adopts depth-separable convolution, which is composed of depth convolution and point-by-point convolution; the built feature extraction layer adds top-down and bottom-up bidirectional cross-layer connections on the basis of FPN, efficiently fuses features of different scales and automatically adjusts the fusion weights according to their importance, so that the deep convolutional neural network can better focus on targets of different scales; the built The detection head layer includes a classification branch and a regression branch. The classification branch determines the target category, and the regression branch predicts the position and size of the target. At the same time, a depth estimation subnetwork and a detection adjustment module are added to the detection head layer. The depth estimation subnetwork uses a convolutional neural network to analyze the features of the input image, learn the relationship between different areas and depth in the image, and generate a depth map to provide a basis for subsequent detection adjustments. The detection adjustment module adaptively adjusts the features of the input detection head based on the depth information obtained by the depth estimation subnetwork. For targets in different depth areas, the feature weights are adjusted or feature transformations are performed to improve the detection head's ability to process target features.
[0013] Step (13): Use the training sample data set established in step (11) to train the deep convolutional neural network built in step (12).
[0014] As a further improvement of the present invention, in the process of extracting features from the harsh image data using the deep convolutional neural network in step 1, the deep convolutional neural network adopts a lightweight design;
[0015] The structure of the deep convolutional neural network uses depthwise separable convolution to replace traditional convolution; specifically, the depthwise separable convolution decomposes the standard convolution into a depthwise convolution step and a point-by-point convolution step, so as to reduce the amount of calculation and the number of parameters in the feature information extraction process.
[0016] As a further improvement of the present invention, the step (13) in the training process of the deep convolutional neural network in step 1 includes:
[0017] Step 13a, pre-training the deep convolutional neural network constructed in step (12);
[0018] Step 13b: Use the severe weather dataset and the secondary expansion dataset in the training sample dataset established in step (11) to train the deep convolutional neural network.
[0019] As a further improvement of the present invention, the use process of the deep convolutional neural network in the step one includes: inputting the harsh image data into the target detection network in the deep convolutional neural network, and the detection head network in the deep convolutional neural network extracting features from the input image data to obtain the features; wherein, the harsh image data is used as input data of the deep convolutional neural network, and the harsh image is input into the deep convolutional neural network to learn drone target detection in harsh weather, so that the deep convolutional neural network can better identify targets in other harsh images when detecting them in the future.
[0020] As a further improvement of the present invention, the improved structure BiFPN in the feature pyramid in the step one adds top-down and bottom-up bidirectional cross-layer connections on the basis of the original feature pyramid structure FPN, so as to more efficiently fuse multi-scale features; and automatically adjusts the fusion weight according to the importance of features of different scales, so as to improve the detection capability of the feature pyramid network for targets of different scales, and further realize the detection capability of targets of different scales with large feature changes in severe weather.
[0021] As a further improvement of the present invention, a depth estimation subnetwork and a detection adjustment module are added to the detection head network in step 2, and the depth estimation subnetwork estimates scene depth information from the input image; specifically, the depth estimation subnetwork performs a layer-by-layer convolution operation on the input image to extract features of different regions; the relationship between different regions in the image and their depths is inferred based on the degree to which the features are affected by severe weather factors, and the scene depth information of the input image is estimated based on the relationship and a comprehensive analysis of the features;
[0022] The detection adjustment module adaptively adjusts the feature map input to the detection head network based on the estimated scene depth information. Specifically, the detection adjustment module divides the image region according to the estimated scene depth information and different types of scene dimensions, and then adopts different adjustment strategies to adjust each image region. The detection adjustment module adjusts the weights of different types of target features based on the image regions of different types of scene dimensions, or performs weighted processing on the feature vectors of target features of different types of scene dimensions according to their locations, and adaptively adjusts the feature map to obtain target adjustment features of different depth regions.
[0023] The detection head network performs weight adjustment or feature transformation on the target adjustment features, so that the detection head network can better focus on and process features related to the target.
[0024] The advantages of the present invention are:
[0025] 1. This paper uses deep learning-based style transfer technology to transfer the style features of real harsh scenes to high-quality images taken in clear weather. It performs geometric transformation on the image data to obtain a double expansion of the harsh weather dataset. In addition, it constructs a GAN model specifically for enhancing harsh weather image data. This allows the present invention to expand the diversity of image data while enhancing the generalization and robustness of the GAN model.
[0026] 2. The present invention obtains target detection information of the image by using a detection head network, which effectively solves the problem of poor target detection effect caused by directly detecting the original image taken in bad weather.
[0027] 3. The present invention enhances the detection performance in severe weather by extracting the depth information of the image instead of directly detecting the original image. It not only enables the network to gradually extract more advanced and abstract features, but also can learn the affected features according to the influence of weather factors, and then infer the relationship between different areas and depth, and conduct a comprehensive analysis of the features of each area in the image, and finally obtain the severe weather UAV target detection information based on the image depth information.
[0028] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:
[0030] Figure 1 : A flow chart of a method for detecting UAV targets in severe weather based on image depth information provided by the present invention;
[0031] Figure 2 : The style transfer model architecture diagram based on the VGG network architecture provided by the present invention;
[0032] Figure 3 : Architecture diagram of the GAN model provided by the present invention;
[0033] Figure 4 : The network structure diagram containing the feature pyramid BiFPN provided by the present invention;
[0034] Figure 5 : The separable convolution DSC Module diagram in the deep convolutional neural network provided by the present invention;
[0035] Figure 6 : A structural diagram of the detection head network provided by the present invention;
[0036] Figure 7:The depth estimation subnetwork and detection adjustment Deep Module diagram provided by the present invention. DETAILED DESCRIPTION
[0037] The following describes in detail embodiments of the present invention. The embodiments are exemplary and intended to explain the present invention, but are not to be construed as limiting the present invention.
[0038] See also Figure 1 The embodiment of the present invention provides a method for detecting UAV targets in severe weather based on image depth information, comprising the following steps a to d:
[0039] Step a: Use style transfer technology to perform style transfer on the clear image data collected under existing clear weather conditions to obtain the severe weather data set required for subsequent detection of targets in the severe image data.
[0040] Due to the current lack of severe weather data, it is impossible to provide an ideal image basis for target detection based on depth information. Therefore, it is necessary to expand the severe weather dataset. In order to meet the needs of expanding the severe weather dataset, the embodiment of the present invention proposes a style transfer model based on the VGG network architecture. Figure 2 The style transfer model shown and Figure 3 The GAN model architecture shown in the figure first extracts the style and texture information of real severe weather images, then fuses this style and texture information into a clear target image to quickly generate training samples with different weather styles. Furthermore, embodiments of the present invention construct a GAN model architecture specifically for severe weather image data enhancement. The generator network in this GAN model architecture is based on a deep convolutional neural network, which generates realistic simulated images by learning the distribution characteristics of real severe weather images.
[0041] The discriminator network in the GAN model architecture of this embodiment of the present invention is responsible for distinguishing generated images from real images, and both are continuously optimized during adversarial training. Using the trained GAN model, virtual image samples with varying weather intensities and combinations of target types are generated based on the needs of actual application scenarios. For example, in foggy scenes, images with varying fog concentrations from light to heavy and containing a variety of targets such as vehicles, pedestrians, and buildings are generated. This further enriches the training dataset and compensates for any deficiencies in the actual collected samples.
[0042] Furthermore, the embodiments of the present invention adopt a more complex and diversified severe weather simulation method, which not only simply adds a fixed degree of influence, but also simulates more realistic and diverse severe weather scenes according to different weather conditions, geographical locations and time changes. For example, in combination with meteorological data, the haze effects of different concentrations, different colors (such as yellow haze, gray haze, etc.), and different distributions (local haze, uniform haze, etc.) are simulated, so that the network can learn a wider range of features and enhance its adaptability to actual complex environments. At the same time, image segmentation technology is combined to annotate the target areas of the migrated and enhanced images to ensure the integrity and recognizability of the target objects in the process of style migration and enhancement, avoid the loss or confusion of target features due to style fusion, and further improve the quality and diversity of the training data set.
[0043] Furthermore, embodiments of the present invention simulate different concentrations of haze by adjusting image pixel transparency, brightness, and other parameters. For example, for high-concentration haze, the overall image brightness can be reduced and the pixel blurriness can be increased to make the image appear hazier. Using a color space conversion algorithm (RGB to HSV), the image hue and saturation are adjusted based on the different haze color characteristics. For yellow haze, the saturation and brightness of the yellow hue are increased; for gray haze, the image's color saturation is reduced and the brightness is adjusted.
[0044] Define different regions on the image and apply different haze simulation parameters to each region. Use a random number generation algorithm, combined with factors such as topography and urban distribution, to simulate different haze distributions. For example, generate localized haze in urban areas and uniform haze in open plains.
[0045] Optionally, embodiments of the present invention overlay raindrop textures on an image to simulate a rainstorm effect. Raindrops of varying sizes, shapes, and transparencies are pre-created, and the density and transparency of the raindrop textures are adjusted. Image filtering algorithms are also used to simulate the blurring effect of raindrops on the scene. Similar to smog simulation, but with different color and graininess of the dust, embodiments of the present invention simulate the dust effect by adjusting the image's color (adding yellow and orange hues) and texture (adding a granular texture). Image clarity and color saturation are adjusted based on the intensity and distance of the dust.
[0046] In actual application, this embodiment uses a style transfer algorithm based on the VGG network in the data preprocessing stage to extract style features from real severe weather images, such as the color distribution of fog and the light and shadow effects of rain scenes. These style features are transferred to clear images, and the content loss Lcontent and style loss Lstyle are jointly optimized. The loss function is L = αLcontent + βLstyle, where α and β are weight coefficients. This embodiment controls the degree of style transfer by adjusting these two coefficients to generate severe weather images with different style intensities. At the same time, this embodiment constructs an improved GAN network. The generator uses depthwise separable convolution to reduce the amount of calculation, and the discriminator introduces a multi-scale feature matching mechanism. During the training process, the gradient penalty term is used to make the generated distribution smoother and avoid mode collapse. The generator and discriminator are updated alternately during the training process, which can generate realistic severe weather images.
[0047] Step b: Use a deep learning model to enhance the severe weather dataset, and use simulation technology to generate a variety of severe weather conditions to obtain a primary expanded dataset; perform small target data enhancement on the primary expanded dataset, and use geometric transformation technology to simulate it in a more diverse manner to obtain a secondary expanded dataset.
[0048] It should be noted that in actual application, since the embodiment of the present invention is for target detection in severe weather from the perspective of a drone, the proportion of small target samples in the dataset is relatively large, and therefore it is necessary to perform data enhancement on the small target samples to improve the accuracy and efficiency of target detection. The embodiment of the present invention uses operations such as random cropping, scaling, and rotation to generate more image samples containing small targets, ensuring the integrity and recognizability of small targets during the enhancement process. At the same time, the embodiment of the present invention increases the contrast between small targets and the background, making small targets more prominent in severe environments and helping the network better learn the characteristics of small targets.
[0049] Step c: Input the bad image data into the trained deep convolutional neural network to extract features, and construct a feature pyramid based on the features; upsample and fuse the features of different layers in the feature pyramid to obtain a feature pyramid with multi-scale information.
[0050] In the above step c of the embodiment of the present invention, a lightweight design is adopted when using a deep convolutional neural network for feature extraction; wherein, the structure of the deep convolutional neural network includes an input layer, a convolution layer, a feature extraction layer and a detection head layer. wherein, the input layer receives the severe weather image data obtained through operations such as style transfer, GAN model enhancement and small target data enhancement; the convolution layer uses depthwise separable convolution to replace the traditional convolution, and the depthwise separable convolution is composed of depthwise convolution and pointwise convolution; the depthwise convolution performs convolution operations on each channel separately, and extracts features only in the spatial dimension, so that each convolution kernel slides only on the corresponding channel, reducing the amount of calculation. Pointwise convolution is a convolution performed in the depth direction, which is used to integrate the feature channels obtained by the depthwise convolution, realize the linear combination of features, and fuse the features of different channels; the feature extraction layer adds top-down and bottom-up bidirectional cross-layer connections on the basis of FPN, which can more efficiently fuse features of different scales. In severe weather conditions, the scale of targets varies greatly. The feature extraction layer automatically adjusts the fusion weights based on the importance of features at different scales, allowing the network to better focus on targets of different scales and improve the detection capabilities of targets of various sizes. The detection head layer includes a classification branch and a regression branch. The classification branch is used to determine the category of the target, and the regression branch is used to predict the position and size of the target. At the same time, a depth estimation subnetwork and a detection adjustment module are added to the detection head. The depth estimation subnetwork uses a convolutional neural network to analyze the features of the input image, learn the relationship between different areas in the image and the depth, and generate a depth map to provide a basis for subsequent detection adjustments. The detection adjustment module obtains depth information based on the depth estimation subnetwork, and adaptively adjusts the features of the input detection head. For targets in different depth areas, it adjusts the feature weights or performs feature transformations to improve the detection head's ability to process target features. In the process of network construction, the embodiment of the present invention selects a pre-trained ResNet network as the basic network, and uses its learned general image features to provide a basis for subsequent training on severe weather images. In the feature extraction stage, this embodiment is designed according to the structure of depth-wise separable convolution, replacing the standard convolution with a combination of depth-wise convolution and point-by-point convolution to achieve a lightweight design and reduce the amount of calculation and parameters. At the same time, the feature outputs of different layers of the base network are connected in the network structure to build a bidirectional cross-layer connection to achieve efficient fusion of multi-scale features. In addition, the detection head constructed in this embodiment is designed with classification branches and regression branches, and a depth estimation subnetwork and detection adjustment module are added to complete the construction of the entire deep convolutional neural network.
[0051] In practical applications, the embodiment of the present invention implements depthwise separable convolution in the convolution layer. For the input feature map X, the depthwise convolution performs convolution operations on each channel separately. If the input feature map has C channels, the depthwise convolution will use C convolution kernels, and each convolution kernel only slides on the corresponding channel. Taking two-dimensional convolution as an example, the calculation process of depthwise convolution is Where Y represents the output feature map, W represents the convolution kernel, M and N are the sizes of the convolution kernel, and k represents the channel index. After the depthwise convolution is completed, the pointwise convolution is performed in the depth direction. The pointwise convolution uses C′ 1×1 convolution kernels to linearly combine the features of the C channels obtained by the depthwise convolution and output the feature map of the C′ channels. The calculation of the pointwise convolution is Where Z is the final output feature map, and V is the weight of the 1×1 convolution kernel. The computational complexity of traditional convolution is C×C′×M×N×H×W (H and W are the height and width of the feature map), while the computational complexity of depthwise separable convolution is C×M×N×H×W+C×C′×H×W, which significantly reduces the computational complexity.
[0052] Specifically, in the feature extraction stage, the embodiment of the present invention adds top-down and bottom-up bidirectional cross-layer connections on the basis of FPN. The top-down path transfers the semantic features of the high-level layer to the low-level layer, and the bottom-up path transfers the detail features of the low-level layer to the high-level layer. Through jump connections, features of different scales can be better fused. At the same time, the attention mechanism is introduced to automatically adjust the fusion weight according to the importance of features of different scales. The attention score of each scale feature is calculated through global average pooling and fully connected layers. For feature maps F1, F2, ..., Fn of different scales, their attention scores A1, A2, ..., An are calculated, and the fused feature maps In this way, the network can better focus on objects of different scales and improve the detection ability of objects of various sizes.
[0053] Specifically, in the detection stage, the embodiment of the present invention uses a convolutional neural network to analyze the features of the input image in the depth estimation subnetwork. Through multi-layer convolution and pooling operations, the high-level semantic features of the image are extracted. During the training process, a large amount of image data with real depth information is used to learn the relationship between different areas of the image and the depth. After the forward propagation of the network, a depth map with the same size as the input image is finally output. Each pixel value in the depth map represents the scene depth corresponding to the position, and then the image is divided into different depth areas, such as close view, mid-range and distant view, based on the depth information obtained by the depth estimation subnetwork. The embodiment of the present invention adjusts the weight of the features for targets in different depth areas. For small targets in the distance, the embodiment of the present invention increases the weight of their features in the feature map, so that the network pays more attention to these targets.
[0054] Furthermore, in the above step c of the embodiment of the present invention, the training process of the deep convolutional neural network includes: (1) pre-training the deep convolutional neural network; (2) training the deep convolutional neural network using the severe weather dataset and the secondary expansion dataset.
[0055] Furthermore, in the above step c of the embodiment of the present invention, the process of using the deep convolutional neural network includes: inputting the harsh image data into the target detection network in the deep convolutional neural network, and the detection head network in the deep convolutional neural network extracts features from the input image data to obtain features; wherein, the harsh image data is used as input data of the deep convolutional neural network, and the harsh image is input into the deep convolutional neural network to learn the drone target detection in harsh weather, so that the deep convolutional neural network can better identify the targets in other harsh images when detecting them in the future.
[0056] Specifically, in step c of the embodiment of the present invention, the improved structure of the feature pyramid network BiFPN adds top-down and bottom-up bidirectional cross-layer connections on the basis of the original feature pyramid structure FPN, and more efficiently fuses multi-scale features; and automatically adjusts the fusion weights according to the importance of features of different scales, so as to improve the detection capability of the feature pyramid network for targets of different scales, and further realize the detection capability of targets of different scales with large feature changes in bad weather.
[0057] After acquiring a relatively rich dataset of severe weather conditions, the present invention uses a progressive training method for a pre-trained ResNet network using this dataset. Specifically, the network is first trained on mild severe weather data, allowing it to initially learn the basic characteristics of the target. The network is then gradually trained on increasing weather severity, allowing the ResNet network to adapt to more complex and severe environments. This training method helps the ResNet network better understand the effects of severe weather on target characteristics, improving its generalization and robustness.
[0058] See also Figure 4 The embodiment of the present invention adopts an improved structure of feature pyramid network BiFPN (Bidirectional Feature Pyramid Network), which is as follows Figure 4 As shown. The improved structure BiFPN of the embodiment of the present invention adds top-down and bottom-up bidirectional cross-layer connections on the basis of FPN, more efficiently fuses multi-scale features, and automatically adjusts the fusion weights according to the importance of features of different scales, further improving the network's detection capabilities for targets of different scales. Since the features of targets of different scales vary greatly in the embodiment of the present invention, especially in bad weather, the above-mentioned improved feature fusion method can improve detection performance. The embodiment of the present invention also adds a context feature extraction module to each level of the feature pyramid, and extracts features of the area around the target through a convolutional neural network. The context feature extraction module captures the context information around the target and fuses the captured context information with the target features. In harsh environments, the environmental information around the target is very important for judging the existence and category of the target.
[0059] Step d: Obtain a feature map based on the feature pyramid, input the feature map into the detection head network, and the detection head network performs regression analysis and positioning analysis on the feature map, and outputs target detection information corresponding to the harsh image data.
[0060] In step d of an embodiment of the present invention, a depth estimation subnetwork and a detection adjustment module are added to the detection head network. The depth estimation subnetwork estimates the scene depth information from the input image, and the detection adjustment module adaptively adjusts the feature map of the input detection head network according to the scene depth information. The detection head network of an embodiment of the present invention adjusts the feature weights or performs feature transformation on targets in different depth areas, so that the detection head network can better focus on and process features related to the target.
[0061] In actual application, after obtaining the feature map, the embodiment of the present invention inputs it into the detection head. In order to further improve the detection head's efficiency in utilizing target features in severe weather, an attention mechanism is used to make the network pay more attention to the features of the target area, so as to reduce the masking of target features by the severe environment, thereby improving the accuracy of detection.
[0062] See also Figure 5 In the embodiment of the present invention, the detection head network structure uses depthwise separable convolution (Depthwise Separable Convolution) to replace the traditional convolution layer in the detection head. Depthwise separable convolution decomposes the standard convolution into two steps: depthwise convolution and point-by-point convolution, which can significantly reduce the amount of calculation and parameters while maintaining good feature extraction capabilities.
[0063] See also Figure 6 The detection head network of the embodiment of the present invention includes a classification branch and a regression branch. In the classification branch, the embodiment of the present invention directly performs a convolution operation on the feature map and outputs the classification result. For a feature map containing multiple candidate areas, the embodiment of the present invention obtains the probability value of each candidate area being a car, pedestrian, bicycle, and other different categories after calculation by the convolution layer. The regression branch is mainly responsible for predicting the position and size information of the target. It also uses the convolution layer to process the input features, calculates the offset of the center coordinates of the candidate area relative to the center coordinates of the true target, and the scaling ratio of the width and height of the candidate area relative to the width and height of the true target.
[0064] See also Figure 7In this embodiment of the present invention, a depth estimation subnetwork and a detection adjustment module are added to the detection head network. The depth estimation subnetwork estimates scene depth information from the input image, analyzes the image's features through a convolutional neural network, and learns the relationship between different image regions and depth, thereby generating a depth map. Specifically, the depth estimation subnetwork performs layer-by-layer convolution operations on the input image to extract features from different regions. It infers the relationship between different image regions and their depth based on the degree to which these features are affected by adverse weather factors. Based on this relationship and a comprehensive analysis of the features, it estimates the scene depth information of the input image and generates a depth map.
[0065] In this embodiment of the present invention, the depth estimation subnetwork in drone imagery captured in inclement weather can capture the correlation between the degree of impact of the inclement environment on objects at different distances and their depth, providing a foundation for subsequent detection adjustments. Specifically, the detection adjustment module adaptively adjusts the feature map input to the detection head network based on the estimated scene depth information. The module divides the image into regions based on the estimated scene depth information and different scene types, and then employs different adjustment strategies to adjust each image region. The module also adjusts the weights of different target features based on the image regions of different scene types, or weights the feature vectors of target features of different scene types based on their locations, adaptively adjusting the feature map to obtain target adjustment features for regions at different depths. More specifically, this embodiment of the present invention performs layer-by-layer convolution on the input image, sliding the convolution kernel across the image to extract features from different image regions. These features include information such as texture, color, and shape. Through multiple layers of convolution, the network is able to gradually extract higher-level, more abstract features. In drone imagery captured in inclement weather, these features can be affected by weather factors. For example, fog can blur the texture of distant objects and change their color. The depth estimation subnetwork learns these affected features to infer the relationship between different image regions and depth. In drone imagery captured in inclement weather, features such as object clarity and color variation are correlated with depth. Distant objects captured using a long-range perspective appear blurrier and have a bluish tint due to environmental influences. The depth estimation subnetwork uses these features to estimate depth, analyzing the edge clarity, detail, and color trends of objects in the image. If an area exhibits blurred edges, loss of detail, and a bluish tint, the depth estimation subnetwork infers that it is farther from the camera and has a larger depth value. Conversely, if an area exhibits sharp edges, rich detail, and vibrant colors, it is closer to the camera and has a smaller depth value. By comprehensively analyzing these features across image regions, the depth estimation subnetwork estimates scene depth information from the input image.
[0066] The present invention analyzes features such as clarity and color variation in an image to infer its relative depth and uses these features to estimate depth information. The detection adjustment module in the present invention adaptively adjusts the features input to the detection head based on the depth information obtained by the depth estimation subnetwork.
[0067] Furthermore, embodiments of the present invention adjust feature weights or perform feature transformations for targets at different depths, enabling the detection head to better focus on and process target-related features. For example, for small, distant targets captured using a distant view in a harsh environment, their feature representation may be enhanced to improve detection accuracy. For nearby targets captured using a close view, which are relatively less affected by the environment, the feature extraction method may be appropriately adjusted to highlight their key features. Specifically, after the depth estimation subnetwork in embodiments of the present invention generates a depth map, the detection adjustment module first divides the image region according to depth values. Based on distance, the depth map is divided into different regions: near view, mid view, and far view. Because targets at different depths are affected differently by inclement weather, embodiments of the present invention employ different adjustment strategies. More specifically, in inclement weather, the features of small, distant targets are often weak in the image and easily masked by ambient noise. Therefore, the detection adjustment module increases the weights of these target features in the feature map, directing the network to focus on them. By evaluating depth information, the region containing the small, distant target is identified, and the feature vectors at the corresponding feature map locations are weighted. The embodiment of the present invention uses an attention mechanism to assign different weight coefficients to different areas based on depth information, giving higher weights to the features of small targets in the distance, thereby strengthening their importance in the subsequent detection process. Nearby targets are relatively less affected by bad weather, but there is interference from details; therefore, the detection adjustment module of the embodiment of the present invention appropriately reduces the weights of these target features to highlight key features. The embodiment of the present invention screens and adjusts the weights of feature maps in the near target area indicated by depth information, reducing the impact of unnecessary detail features, avoiding excessive interference to the network during detection, and improving detection accuracy.
[0068] The embodiments of the present invention use deep learning-based style transfer technology to transfer the style features of real harsh scenes to high-quality images taken in clear weather. The image data is geometrically transformed and a GAN model specifically designed for enhancing image data in harsh weather is constructed. This can not only expand the diversity of image data, but also enhance the generalization and robustness of the GAN model.
[0069] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.
Claims
1. A method for detecting UAV targets in severe weather based on image depth information, characterized in that: include: Step 1: Input the severe image data collected under severe weather conditions into the trained deep convolutional neural network to extract features, and construct a feature pyramid based on the features; Upsampling and fusing features at different layers in the feature pyramid to obtain a feature pyramid with multi-scale information; Step 2: obtaining a feature map based on the feature pyramid, inputting the feature map into a detection head network, and outputting target detection information corresponding to the bad image data after performing regression analysis and positioning analysis on the feature map; Specifically, the training process of the deep convolutional neural network in step 1 includes: Step (11), establishing a training sample data set containing severe weather data; wherein the severe weather data in the training sample data set is obtained by the following process: step a, using style transfer technology to perform style transfer on clear image data collected under existing clear weather conditions to obtain an initial severe weather data set; step b, using a deep learning model to enhance the severe weather data set, and using simulation technology to generate a variety of severe weather conditions to obtain a primary expansion data set; performing small target data enhancement on the primary expansion data set, and using geometric transformation technology to perform a more diverse simulation on it to obtain a secondary expansion data set; Step (12), building a deep convolutional neural network; specifically, building an input layer, a convolutional layer, a feature extraction layer and a detection head layer in sequence, the built input layer receives the bad weather image data obtained through style transfer, GAN model enhancement and small target data enhancement operations; the built convolutional layer adopts depth-separable convolution, which is composed of depth convolution and point-by-point convolution; the built feature extraction layer adds top-down and bottom-up bidirectional cross-layer connections on the basis of FPN, efficiently fuses features of different scales and automatically adjusts the fusion weights according to their importance, so that the deep convolutional neural network can better focus on targets of different scales; the built The detection head layer includes a classification branch and a regression branch. The classification branch determines the target category, and the regression branch predicts the position and size of the target. At the same time, a depth estimation subnetwork and a detection adjustment module are added to the detection head layer. The depth estimation subnetwork uses a convolutional neural network to analyze the features of the input image, learn the relationship between different areas and depth in the image, and generate a depth map to provide a basis for subsequent detection adjustments. The detection adjustment module adaptively adjusts the features of the input detection head based on the depth information obtained by the depth estimation subnetwork. For targets in different depth areas, the feature weights are adjusted or feature transformations are performed to improve the detection head's ability to process target features. Step (13): Use the training sample data set established in step (11) to train the deep convolutional neural network built in step (12).
2. The method for detecting UAV targets in severe weather based on image depth information according to claim 1, characterized in that: In the step 1, in which the deep convolutional neural network is used to extract features from the harsh image data, the deep convolutional neural network adopts a lightweight design; depthwise separable convolution is used in the structure of the deep convolutional neural network to replace traditional convolution; specifically, the depthwise separable convolution decomposes the standard convolution into a depthwise convolution step and a point-by-point convolution step, so as to reduce the amount of calculation and the amount of parameters in the feature information extraction process.
3. The method for detecting UAV targets in severe weather based on image depth information according to claim 1, characterized in that: The deep convolutional neural network in step 1, wherein step (13) in the training process includes: Step (13a), pre-training the deep convolutional neural network constructed in step (12); Step (13b): training the deep convolutional neural network using the severe weather dataset and the secondary expansion dataset in the training sample dataset established in step (11).
4. The method for detecting UAV targets in severe weather based on image depth information according to claim 3, characterized in that: The deep convolutional neural network in step 1 is used in the following process: inputting the harsh image data into the target detection network in the deep convolutional neural network, and the detection head network in the deep convolutional neural network extracting features from the input image data to obtain the features; wherein, the harsh image data is used as input data of the deep convolutional neural network, and the harsh image is input into the deep convolutional neural network to learn the detection of drone targets in severe weather, so that the deep convolutional neural network can better identify targets when detecting other harsh images in the future.
5. The method for detecting UAV targets in severe weather based on image depth information according to claim 1, characterized in that: The improved structure BiFPN in the feature pyramid in step 1 adds top-down and bottom-up bidirectional cross-layer connections on the basis of the original feature pyramid structure FPN, so as to more efficiently fuse multi-scale features; And automatically adjust the fusion weight according to the importance of features at different scales to improve the detection capability of the feature pyramid network for targets of different scales, thereby achieving the detection capability of targets of different scales with large feature changes in severe weather.
6. The method for detecting UAV targets in severe weather based on image depth information according to claim 1, characterized in that: A depth estimation subnetwork and a detection adjustment module are added to the detection head network in step 2, wherein the depth estimation subnetwork estimates scene depth information from the input image. Specifically, the depth estimation subnetwork performs a layer-by-layer convolution operation on the input image to extract features of different regions. The relationship between different regions in the image and their depths is inferred based on the degree to which the features are affected by severe weather factors, and the scene depth information of the input image is estimated based on the relationship and a comprehensive analysis of the features. The detection adjustment module adaptively adjusts the feature map input to the detection head network based on the estimated scene depth information. Specifically, the detection adjustment module divides the image region according to the estimated scene depth information and different types of scene dimensions, and then adopts different adjustment strategies to adjust each image region. The detection adjustment module adjusts the weights of different types of target features based on the image regions of different types of scene dimensions, or performs weighted processing on the feature vectors of target features of different types of scene dimensions according to their locations, and adaptively adjusts the feature map to obtain target adjustment features of different depth regions. The detection head network performs weight adjustment or feature transformation on the target adjustment features, so that the detection head network can better focus on and process features related to the target.