Vehicle rotation detection method and system from the perspective of unmanned aerial vehicle

Abstract

The invention belongs to the field of rotation target detection from the perspective of an unmanned aerial vehicle in computer vision, and specifically relates to a vehicle rotation detection method and system from the perspective of an unmanned aerial vehicle; the invention comprises an image acquisition module, an image label processing module, an image preprocessing module, a backbone module, a feature space relationship capture module, a first feature fusion module, a second feature fusion module, a CSL annular smoothing processing module and a prediction module; the feature space relationship capture module is connected to the backbone module and the first feature fusion module; the first feature fusion module and the second feature fusion module are connected by using a level 1 channel, a level 2 channel and a level 3 channel; the method and system of the invention rely on an unmanned aerial vehicle and a visual sensor thereof to acquire image data, and have low hardware requirements; spatial relationship information and feature information extracted by the backbone are fused in two ways, namely, fixed weight and adaptive weight, to predict the position, angle and category information of the vehicle, and to realize vehicle monitoring from the perspective of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention belongs to the field of rotating target detection from the perspective of an unmanned aerial vehicle in computer vision, and in particular relates to a vehicle rotation detection method and system from the perspective of an unmanned aerial vehicle. Background Art

[0002] General object detection algorithms are mainly used to detect general objects in images with horizontal bounding boxes. Among them, in anchor-based object detection algorithms, typical ones are two-stage algorithms such as RCNN and Faster-RCNN and one-stage algorithms such as SSD, YOLO, and RetinaNet. However, these general object detectors will have the following three problems when directly applied to vehicle images or videos taken by drones:

[0003] 1. General target detection algorithms cannot provide angle information. General target detection algorithms predict horizontal detection frames and cannot predict the angle information of the target. This will cause many problems when used in scenarios where angle information is required, such as drone or remote sensing image target detection.

[0004] 2. The general target detector is often poorly applied to a specific scene. The scene targeted by the present invention is the vehicle pictures and videos taken by the drone at an altitude of 30-80m from a top-down perspective. This scene is very different from the natural perspective or the perspective of the surveillance camera, and the distribution of vehicles in this scene itself has distinct characteristics. However, there is no rotating target detector specifically designed for this scene, and the general rotating target detector performs poorly in this scene.

[0005] 3. Most general-purpose rotating target detectors have large models and cannot perform real-time detection. Especially those rotating target detectors based on two-stage general-purpose target detectors, the models are often large and complex, and each inference requires a lot of computing resources, which cannot perform real-time detection. In order to ensure the detection performance, many rotating target detectors based on one-stage general-purpose target detectors are often designed with large models, which cannot be applied to drones.

[0006] The application direction of the present invention is unmanned aerial vehicles, so the model of the basic detector cannot be very large, and the detection speed must reach real-time, so a two-stage detector cannot be selected; in the one-stage detector, considering the comprehensive detection performance, detection speed and model size, the YOLO series detector is selected, especially the 5th version of the series, which has 3 models of different sizes to choose from, namely YOLOv5s (small), YOLOv5m (medium) and YOLOv5l (large). Taking into account the detection speed and the size of the model, YOLOv5s is selected as the basic detector. On this basis, a vehicle rotation detection model is designed for the specific scenario of a vehicle photographed by a drone at an altitude of 30-80m from a top-down perspective. Summary of the invention

[0007] In order to solve the above problems, the present invention provides a vehicle rotation detection system from the perspective of a drone, comprising an image acquisition module, an image label processing module, an image preprocessing module, a Backbone module, a feature space relationship capture module, a first feature fusion module, a second feature fusion module, a CSL annular smoothing processing module and a prediction module, wherein the feature space relationship capture module is connected to the Backbone module and the first feature fusion module, and the first feature fusion module and the second feature fusion module are connected using a level 1 channel, a level 2 channel and a level 3 channel, wherein:

[0008] Image acquisition module, used to acquire images taken by the drone at an altitude of 30-80m from a top-down perspective;

[0009] The image label processing module is used to process the label data of the image, use the long edge representation method to represent the label of each target in the image, and combine the image data obtained by the image acquisition module to build the system training data set;

[0010] An image preprocessing module, used for preprocessing the image in the image acquisition module;

[0011] Backbone module, used to extract the underlying features of the image output by the image preprocessing module;

[0012] The feature spatial relationship capture module is used to obtain the spatial relationship between feature points in the feature map extracted at 16 times downsampling and 32 times downsampling in the Backbone module, and then capture the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment;

[0013] The first feature fusion module is used to extract the feature map at the 8-fold downsampling of the Backbone module and the feature map in the feature space relationship capture module, perform up- and down-sampling and fusion, and output the fused different feature maps to the second feature fusion module through three channels of level 1, level 2 and level 3 respectively;

[0014] The second feature fusion module is used to sample and fuse the different fused features output by the first feature fusion module in the level 1 channel, the level 2 channel and the level 3 channel according to the adaptive spatial feature fusion, and each channel outputs a corresponding feature map after fusion;

[0015] CSL annular smoothing processing module, used for the system to use CSL algorithm to perform annular smoothing on the angle information in the tag in training mode, so that the angle information in the tag has periodicity, annular characteristics and local smoothness characteristics;

[0016] The prediction module is used to predict the position information, angle information, and category information of the vehicle target in the image, and output the corresponding prediction confidence.

[0017] Furthermore, the long edge representation is used in the image label processing module to represent the label of each target in the image. The format of the label after processing is:

[0018] [classid,x_c,y_c,longside,shortside,θ]

[0019] Where classid is the category information label, (x_c, y_c) is the coordinate of the center point of the target, longside is the longest side of the rotated rectangle, shortside is the other side of the rotated rectangle corresponding to the longest side, and θ is the angle of the x-axis when it rotates clockwise to meet the longest side, and the value range is an integer from 0° to 179°.

[0020] Furthermore, the Backbone module includes:

[0021] The Focus layer, the first convolution layer, the first CSP bottleneck layer, the second convolution layer, the second CSP bottleneck layer, the third convolution layer, the third CSP bottleneck layer, the fourth convolution layer, and the SPP layer are cascaded in sequence;

[0022] The Focus layer divides the image processed by the image preprocessing module into four parts. The depth of each part remains unchanged, and the length and width are halved. Then, the parts are spliced ​​in the depth dimension and finally convolution is performed.

[0023] The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer are used to perform a convolution operation of downsampling the input image by 2 times;

[0024] The first CSP bottleneck layer, the second CSP bottleneck layer, and the third CSP bottleneck layer perform convolution operations on the input image;

[0025] The SPP layer is used to perform the maximum pooling operation according to the maximum pooling layer with a stride of 1 and a window size of 5×5, 9×9, and 13×13, and then concatenate in the depth direction, and finally adjust the number of channels of the feature map through a 1×1 convolution.

[0026] Further, the feature space relationship capture module includes a first Bottleneck Transformer layer and a second Bottleneck Transformer layer;

[0027] The first Bottleneck Transformer layer is used to capture the spatial relationship between feature points of the feature map extracted at 32 times downsampling in the Backbone module, thereby capturing the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment;

[0028] The second Bottleneck Transformer layer is used to capture the spatial relationship between feature points of the feature map extracted at 16 times downsampling in the Backbone module, and then capture the spatial relationship between vehicles and the spatial relationship between vehicles and the surrounding environment in the original image.

[0029] Furthermore, the first feature fusion module includes:

[0030] The fifth convolutional layer, the first upsampling layer, the first splicing layer, the sixth convolutional layer, the second upsampling layer, the second splicing layer, the fourth CSP bottleneck layer, the seventh convolutional layer, the third splicing layer, the fifth CSP bottleneck layer, the eighth convolutional layer, the fourth splicing layer, and the sixth CSP bottleneck layer are cascaded in sequence;

[0031] The first concatenation layer is used to concatenate the feature map of the second Bottleneck Transformer layer and the feature map of the first upsampling layer;

[0032] The second splicing layer is used to splice the feature map of the second CSP bottleneck layer and the feature map of the second upsampling layer;

[0033] The third concatenation layer is used to concatenate the feature map of the sixth convolutional layer and the feature map of the seventh convolutional layer;

[0034] The fourth concatenation layer is used to concatenate the feature map of the fifth convolution layer and the feature map of the eighth convolution layer;

[0035] The fifth convolution layer and the sixth convolution layer are used to perform convolution operations on the feature map to adjust the number of feature map channels;

[0036] The seventh and eighth convolutional layers are used to perform convolution operations on the feature maps with a 2x downsampling;

[0037] The fourth CSP bottleneck layer extracts features from the input image and outputs a feature map with a depth of 128 and a width and height that is 8 times the downsampled size of the image obtained by the image acquisition module to the level 1 channel;

[0038] The fifth CSP bottleneck layer extracts features from the input image and outputs a feature map with a depth of 256 and a width and height that is 16 times the downsampled size of the image obtained by the image acquisition module to the level 2 channel;

[0039] The sixth CSP bottleneck layer extracts features from the input image and outputs a feature map with a depth of 512 and a width and height that is 32 times the downsampled size of the image acquired by the image acquisition module to the level3 channel.

[0040] Furthermore, the second feature fusion module adjusts the three feature maps output by the first feature fusion module by constant scaling, so that the sizes and the number of channels of the three feature maps are the same when different channels are fused. The constant scaling specifically includes:

[0041] When the level 1 channel fuses the feature maps output by level 2 and level 3, the feature map output by level 2 is upsampled by 2 times, and the feature map output by level 3 is upsampled by 4 times;

[0042] When the level 2 channel fuses the feature maps output by level 1 and level 3, the feature map output by level 1 is upsampled by 2 times, and the feature map output by level 3 is downsampled by 2 times using a 3×3 convolution with a stride of 2;

[0043] When the level 3 channel fuses the feature maps output by level 1 and level 2, the feature map output by level 1 is first downsampled by 2 using a 3×3 convolution with a stride of 2, and then downsampled by 2 using a 3×3 maximum pooling with a stride of 2. The feature map output by level 2 is downsampled by 2 using a 3×3 convolution with a stride of 2.

[0044] Furthermore, the second feature fusion module uses constant scaling to perform adaptive fusion and fuses the adjusted feature maps. The adaptive fusion is expressed as:

[0045]

[0046] in, represents the feature map after adaptive fusion, (i, j) represents the spatial position after the fusion of the three feature maps, It represents the spatial importance parameter learned by the feature map output by the level 1 channel after 1×1 convolution. It represents the spatial importance parameter learned after 1×1 convolution of the feature map output by level 2 channel. It represents the spatial importance parameter learned after 1×1 convolution of the feature map output by level 3 channel. Represents the feature at position (i, j) in the level 1 channel, Represents the feature at position (i, j) in the level 2 channel, Represents the feature at position (i, j) in the level 3 channel.

[0047] Furthermore, the CSL algorithm used in the CSL ring smoothing processing module is expressed as:

[0048]

[0049] Among them, θ represents the angle value in the current target label in the training mode, r represents the window radius, and its value is 10; g(x) represents the Gaussian function, and its expression is:

[0050]

[0051] Among them, μ represents the mean, which is 0; σ represents the variance, which is 10.

[0052] Furthermore, the prediction module includes three prediction feature layers, namely, the ninth convolutional layer, the tenth convolutional layer, the eleventh convolutional layer, the small layer, the medium layer and the large layer, where:

[0053] The input of the ninth convolutional layer is the feature map output by the level 1 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point.

[0054] The input of the tenth convolutional layer is the feature map output by the level 2 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point.

[0055] The input of the eleventh convolutional layer is the feature map output by the level 3 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point.

[0056] The ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer have the same number of convolutions, and the number of convolutions = xywh + confidence + num_class + num_angle.

[0057] Among them, xywh is the coordinate of the center point of the target on the image and the offset of the width and height of the target on the image relative to the initial anchor box, confidence refers to the confidence, num_class is the category of the target, num_angle is the category of the angle, and after using the CSL algorithm, num_angle takes an integer in the range of 0°-179°;

[0058] The small layer is the feature map output by the ninth convolutional layer, which is used to predict small objects in the image;

[0059] The medium layer is the feature map output by the tenth convolutional layer, which is used to predict medium-sized objects in the image;

[0060] The large layer is the feature map output by the eleventh convolutional layer and is used to predict large objects in the image.

[0061] A method for detecting vehicle rotation from a drone's perspective comprises the following steps:

[0062] The image acquisition module is used to obtain images taken by the drone at an altitude of 30-80m from a direct viewpoint.

[0063] Preprocessing the image in the image acquisition module through the image preprocessing module;

[0064] The Backbone module is used to extract the underlying features of the image output by the image preprocessing module;

[0065] The feature spatial relationship capture module is used to obtain the spatial relationship between feature points in the feature map extracted by the Backbone module at 16x downsampling and 32x downsampling, thereby capturing the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment.

[0066] The first feature fusion module is used to fuse the features extracted from the 8-fold downsampling of the Backbone module and the features extracted from the feature space relationship capture module to obtain the fused features.

[0067] The second feature fusion module is used to perform a second feature fusion on the fused features and output three feature maps of different sizes;

[0068] The prediction module is used to predict the position information, angle information, and category information of the vehicle target in the image, and output the corresponding prediction confidence.

[0069] Beneficial effects of the present invention:

[0070] The present invention can detect targets at any angle. Compared with general target detection, the present invention can not only predict the position information, category information and confidence information of the target on the image, but also predict the angle information of the target, which greatly increases the application field and scope.

[0071] By constructing a feature spatial relationship capture module, the model can effectively capture the spatial relationship between feature points, and then capture the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment, effectively improving the model's performance in detecting vehicle targets in drone scenarios.

[0072] By connecting the second feature fusion module ASFF after the first feature fusion module PANet, the effect of feature fusion is further enhanced, effectively improving the detection performance of the system in scenes with drastic changes in target scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0074] Figure 2 A schematic diagram showing the spatial correspondence between the features extracted by Backbone of the present invention and the original image;

[0075] Figure 3 It is a schematic diagram of applying the self-attention mechanism of the present invention to an image feature map. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] A vehicle rotation detection system from the perspective of a drone, such as Figure 1 As shown, it includes an image acquisition module, an image label processing module, an image preprocessing module, a Backbone module, a feature space relationship capture module, a first feature fusion module, a second feature fusion module, a CSL annular smoothing processing module and a prediction module. The feature space relationship capture module connects the Backbone module and the first feature fusion module. The first feature fusion module and the second feature fusion module are connected using level 1 channels, level 2 channels and level 3 channels, wherein:

[0078] Image acquisition module, used to obtain vehicle pictures and videos taken by drones at an altitude of 30-80m from a top-down perspective;

[0079] The image label processing module is used to process the label data of the image, use the long edge representation method to represent the label of each target in the image, and combine the image data obtained by the image acquisition module to build the system training data set;

[0080] An image preprocessing module, used for preprocessing the image in the image acquisition module;

[0081] Backbone module, used to extract the underlying features of the image output by the image preprocessing module;

[0082] The feature spatial relationship capture module is used to obtain the spatial relationship between feature points in the feature map extracted at 16 times downsampling and 32 times downsampling in the Backbone module, and then capture the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment, thereby improving the prediction performance of the system;

[0083] The first feature fusion module is used to extract the feature map at the 8-fold downsampling of the Backbone module and the feature map in the feature space relationship capture module, perform up- and down-sampling and fusion, and output the fused different feature maps to the second feature fusion module through three channels of level 1, level 2 and level 3 respectively;

[0084] The second feature fusion module is used to sample and fuse the different fused feature maps output by the first feature fusion module in the level 1 channel, the level 2 channel and the level 3 channel according to the adaptive spatial feature fusion, and each channel outputs a corresponding feature map after fusion;

[0085] CSL annular smoothing processing module, used for the system to use CSL algorithm to perform annular smoothing on the angle information in the tag in training mode, so that the angle information in the tag has periodicity, annular characteristics and local smoothness characteristics;

[0086] The prediction module is used to predict the position information, angle information, and category information of the vehicle target in the image, and output the corresponding prediction confidence.

[0087] Specifically, Figure 1 As shown, the Backbone module has a total of 9 layers, including:

[0088] The 0th layer is the Focus layer, which is used to divide the image processed by the image preprocessing module into 4 parts. The depth of each data part remains unchanged, the length and width are halved, and then they are spliced ​​in the depth dimension, and finally the convolution operation is performed;

[0089] The first, third, fifth and seventh layers are all conv layers, corresponding to the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer, respectively, and are all used to perform a convolution operation of downsampling the input image by a factor of 2;

[0090] The second layer is C3_1, the first CSP bottleneck layer, which has 1 Bottleneck. This layer is used to perform convolution operations on the input image.

[0091] The 4th and 6th layers are C3_3 layers, i.e., the second CSP bottleneck layer and the third CSP bottleneck layer, both of which have 3 bottlenecks. These two layers are used to perform convolution operations on the input image.

[0092] The 8th layer is the SPP layer, which is used to perform the maximum pooling operation according to the maximum pooling layer with a step size of 1 and a window size of 5×5, 9×9, and 13×13, and then splice in the depth direction, and finally adjust the number of channels of the feature map through a 1×1 convolution.

[0093] Preferably, if Figure 1 As shown, the feature space relationship capture module includes the first Bottleneck Transformer layer and the second Bottleneck Transformer layer, i.e., the 9th CTR3 layer and the 12th CTR3 layer;

[0094] The first Bottleneck Transformer layer is used to capture the spatial relationship between feature points of the feature map extracted at 32 times downsampling in the Backbone module, thereby capturing the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment;

[0095] The second Bottleneck Transformer layer is used to capture the spatial relationship between feature points of the feature map extracted at 16 times downsampling in the Backbone module, and then capture the spatial relationship between vehicles and the spatial relationship between vehicles and the surrounding environment in the original image.

[0096] Preferably, if Figure 1 As shown, the first feature fusion module includes:

[0097] The 10th, 14th, 18th, and 21st layers are all conv layers, corresponding to the fifth, sixth, seventh, and eighth convolutional layers, respectively. The fifth and sixth convolutional layers are used to perform convolution operations on feature maps to adjust the number of feature map channels; the seventh and eighth convolutional layers are used to perform convolution operations on feature maps to downsample by 2 times;

[0098] The 11th and 15th layers are both upsampling layers, i.e. the first upsampling layer and the second upsampling layer;

[0099] The 12th layer is the concat layer, i.e. the first concatenation layer, which is used to concatenate the feature map of the second Bottleneck Transformer layer and the feature map of the first upsampling layer;

[0100] The 16th layer is the concat layer, i.e. the second concatenation layer, which is used to concatenate the feature map of the second CSP bottleneck layer and the feature map of the second upsampling layer;

[0101] The 19th layer is the concat layer, i.e. the third concatenation layer, which is used to concatenate the feature map of the sixth convolutional layer and the feature map of the seventh convolutional layer;

[0102] The 22nd layer is the concat layer, i.e. the fourth concatenation layer, which is used to concatenate the feature map of the fifth convolutional layer and the feature map of the eighth convolutional layer;

[0103] The 17th layer is C3_1 layer, which is the fourth CSP bottleneck layer. It has 1 Bottleneck. This layer is used to extract features from the input image and output a feature map with a depth of 128 and a width and height that is 8 times the downsampled size of the image obtained by the image acquisition module to the level 1 channel.

[0104] The 20th layer is C3_1 layer, which is the fifth CSP bottleneck layer. It has 1 Bottleneck. This layer is used to extract features from the input image and output a feature map with a depth of 256 and a width and height that is 16 times the downsampled size of the image obtained by the image acquisition module to the level2 channel.

[0105] The 23rd layer is C3_1 layer, which is the sixth CSP bottleneck layer. It has 1 Bottleneck. This layer extracts features from the input image and outputs a feature map with a depth of 512 and a width and height that is 32 times the downsampled size of the image obtained by the image acquisition module to the level3 channel.

[0106] In one embodiment, the feature space relationship capture module obtains the feature maps of the 6th and 8th layers in the Backbone module, with sizes of 256×60×60, where 256 is the depth information of the feature map, that is, the channel is 256, and 60×60 is the width and height of the feature map; 512×30×30, where 512 is the depth information of the feature map, that is, the channel is 512, and 30×30 is the width and height of the feature map;

[0107] The first feature fusion module extracts the feature map of the 4th layer in the Backbone module, whose size is 128×120×120, where 128 is the depth information of the feature map, that is, the channel is 128, and 120×120 is the width and height of the feature map, and the two feature maps in the feature spatial relationship capture module. These three feature maps are upsampled and downsampled and then fused.

[0108] In one embodiment, if Figure 1 As shown in the figure, the first feature fusion module, namely the PANet feature fusion, is a fixed-weight feature fusion. The sizes of the two feature maps are exactly the same when spliced. However, for the images taken by the drone flying up and down, the image scale changes dramatically and uncertainly. Therefore, only fixed-weight feature fusion is not enough, and adaptive weight feature fusion is also required. Therefore, the second feature fusion module, namely the ASFF adaptive spatial feature fusion module, is connected after PANet. The fusion layer is used to perform adaptive spatial feature fusion on the three feature maps 128×120×120, 256×60×60, and 512×30×30 output by PANet to further enhance the effect of feature fusion.

[0109] Specifically, Figure 3 As shown in , the feature map of an image can also be considered as a set of vectors. For example, for a 256×30×30 feature map, each vector is 256×1×1, so there are 30×30 vectors in total. Some of these vectors represent vehicle information on the original image, some vectors represent some background information on the original image that is more related to the vehicle, such as roads, parking lots, etc., and some vectors represent some background information that is less related to the vehicle, such as houses, etc. Therefore, by using the self-attention mechanism on this feature map to capture the relationship between vectors, the relationship between vehicles on the original image and the relationship between vehicles and environments that are more related, such as roads, can be modeled. These can help the model better detect vehicle targets, because the distribution of vehicles is regular, and it is easier to find another vehicle around a vehicle. At the same time, vehicles will only appear in scenes such as roads and parking lots, but not in houses.

[0110] Preferably, the size of the feature map of the 8th layer is 512×30×30, and the size of the feature map of the 6th layer is 256×60×60. The resolution of high-definition images taken by drones is generally 3×1920×1080 and 3×2720×1530. After padding these two images and adjusting them to 3×960×960, if there is a vehicle in the original image, the size of the vehicle on the 3×960×960 image is about 30×70 (the original image is 1920×1080) or 20×40 (the original image is 2720×1530). After feature extraction, feature maps of 512×30×30 and 256×60×60 are obtained. One or two feature points on these two feature maps can represent a vehicle at the corresponding position of the original image. Therefore, the use of the self-attention mechanism on these two feature maps can effectively capture the relationship between each feature point on this feature map relative to other feature points, and the features of some feature points are the features of the corresponding position of the vehicle on the original image. Therefore, by applying the self-attention mechanism on the feature map of this size, the spatial relationship between the vehicles in the original image and the spatial relationship between the vehicle and the environment can be effectively captured, such as Figure 2 shown.

[0111] In one embodiment, the second feature fusion module adjusts the three feature maps output by the first feature fusion module by constant scaling, so that the size and number of channels of the three feature maps are the same when different channels are fused, and the constant scaling specifically includes:

[0112] When the level 1 channel fuses the feature maps output by level 2 and level 3, the feature map output by level 2 is upsampled by 2 times, and the feature map output by level 3 is upsampled by 4 times;

[0113] When the level 2 channel fuses the feature maps output by level 1 and level 3, the feature map output by level 1 is upsampled by 2 times, and the feature map output by level 3 is downsampled by 2 times using a 3×3 convolution with a stride of 2;

[0114] When the level 3 channel fuses the feature maps output by level 1 and level 2, the feature map output by level 1 is first downsampled by 2 using a 3×3 convolution with a stride of 2, and then downsampled by 2 using a 3×3 maximum pooling with a stride of 2. The feature map output by level 2 is downsampled by 2 using a 3×3 convolution with a stride of 2.

[0115] Preferably, the second feature fusion module performs adaptive fusion after constant scaling to fuse the adjusted feature maps. The adaptive fusion is expressed as:

[0116]

[0117] in, represents the feature map after adaptive fusion, (i, j) represents the spatial position after the fusion of the three feature maps, It represents the spatial importance parameter learned by the feature map output by the level 1 channel after 1×1 convolution. It represents the spatial importance parameter learned after 1×1 convolution of the feature map output by level 2 channel. It represents the spatial importance parameter learned after 1×1 convolution of the feature map output by level 3 channel. Represents the feature at position (i, j) in the level 1 channel, Represents the feature at position (i, j) in the level 2 channel, Represents the feature at position (i, j) in the level 3 channel.

[0118] In one embodiment, the prediction module includes three prediction feature layers, namely, a ninth convolutional layer, a tenth convolutional layer, an eleventh convolutional layer, a small layer, a medium layer, and a large layer, wherein:

[0119] The input of the ninth convolutional layer is the feature map output by the level 1 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point.

[0120] The input of the tenth convolutional layer is the feature map output by the level 2 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point.

[0121] The input of the eleventh convolutional layer is the feature map output by the level 3 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point.

[0122] The ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer have the same number of convolutions, and the number of convolutions = xywh + confidence + num_class + num_angle.

[0123] Among them, xywh is the coordinate of the center point of the target on the image and the offset of the width and height of the target on the image relative to the initial anchor box, confidence refers to the confidence, num_class is the category of the target, num_angle is the category of the angle, and after using the CSL algorithm, num_angle takes an integer in the range of 0°-179°;

[0124] The small layer is the feature map output by the ninth convolutional layer, which is used to predict small objects in the image;

[0125] The medium layer is the feature map output by the tenth convolutional layer, which is used to predict medium-sized objects in the image;

[0126] The large layer is the feature map output by the eleventh convolutional layer and is used to predict large objects in the image.

[0127] Preferably, the long edge representation is used in the image label processing module to represent the label of each object in the image, and the format of the label after processing is:

[0128] [classid,x_c,y_c,longside,shortside,θ]

[0129] Where classid is the category information id, (x_c, y_c) is the coordinate of the center point of the target, longside is the longest side of the rotated rectangle, shortside is the other side of the rotated rectangle corresponding to the longest side, and θ is the angle of the x-axis when it rotates clockwise to meet the longest side, and the value range is an integer from 0° to 179°.

[0130] Preferably, the CSL annular smoothing processing module is used for the system to use the CSL algorithm to perform annular smoothing on the angle information in the tag in the training mode. The CSL algorithm used in the module is expressed as:

[0131]

[0132] Among them, θ represents the angle value in the current target label in the training mode, r represents the window radius, and its value is 10; g(x) represents the Gaussian function, and its expression is:

[0133]

[0134] Among them, μ represents the mean, which is 0; σ represents the variance, which is 10.

[0135] A method for detecting vehicle rotation from a drone's perspective comprises the following steps:

[0136] The image acquisition module is used to obtain images taken by the drone at an altitude of 30-80m from a direct viewpoint.

[0137] Preprocessing the image in the image acquisition module through the image preprocessing module;

[0138] The Backbone module is used to extract the underlying features of the image output by the image preprocessing module;

[0139] The feature spatial relationship capture module is used to obtain the spatial relationship between feature points in the feature map extracted by the Backbone module at 16x downsampling and 32x downsampling, thereby capturing the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment.

[0140] The first feature fusion module is used to fuse the features extracted from the 8-fold downsampling of the Backbone module and the features extracted from the feature space relationship capture module to obtain the fused features.

[0141] The second feature fusion module is used to perform a second feature fusion on the fused features and output three feature maps of different sizes;

[0142] The prediction module is used to predict the position information, angle information, and category information of the vehicle target in the image, and output the corresponding prediction confidence.

[0143] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle rotation detection system from the perspective of a drone, characterized in that: It includes an image acquisition module, an image label processing module, an image preprocessing module, a Backbone module, a feature space relationship capture module, a first feature fusion module, a second feature fusion module, a CSL annular smoothing processing module and a prediction module. The feature space relationship capture module connects the Backbone module and the first feature fusion module. The first feature fusion module and the second feature fusion module are connected using level 1 channels, level 2 channels and level 3 channels, wherein: Image acquisition module, used to acquire images taken by the drone at an altitude of 30-80m from a top-down perspective; The image label processing module is used to process the label data of the image, use the long edge representation method to represent the label of each target in the image, and combine the image data obtained by the image acquisition module to build the system training data set; An image preprocessing module, used for preprocessing the image in the image acquisition module; Backbone module, used to extract the underlying features of the image output by the image preprocessing module; The feature spatial relationship capture module is used to obtain the spatial relationship between feature points in the feature map extracted at 16 times downsampling and 32 times downsampling in the Backbone module, and then capture the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment; The first feature fusion module is used to extract the feature map of the 8-fold downsampling of the Backbone module and the feature map in the feature space relationship capture module for up- and down-sampling fusion, and output the fused different feature maps to the second feature fusion module through three channels of level 1, level 2 and level 3 respectively; The second feature fusion module is used to sample and fuse the different fused feature maps output by the first feature fusion module in the level 1 channel, the level 2 channel and the level 3 channel according to the adaptive spatial feature fusion, and each channel outputs a corresponding feature map after fusion; CSL annular smoothing processing module, used for the system to use CSL algorithm to perform annular smoothing on the angle information in the tag in training mode; The prediction module is used to predict the position information, angle information, and category information of the vehicle target in the image, and output the corresponding prediction confidence.

2. The vehicle rotation detection system from the perspective of an unmanned aerial vehicle according to claim 1, characterized in that: The image label processing module uses the long edge representation to represent the label of each target in the image. The format of the label after processing is: [classid,x_c,y_c,longside,shortside,θ] Where classid is the category information label, (x_c, y_c) is the coordinate of the center point of the target, longside is the longest side of the rotated rectangle, shortside is the other side of the rotated rectangle corresponding to the longest side, and θ is the angle of the x-axis when it rotates clockwise to meet the longest side, and the value range is an integer from 0° to 179°.

3. The vehicle rotation detection system from the perspective of an unmanned aerial vehicle according to claim 1, characterized in that: Backbone modules include: The Focus layer, the first convolution layer, the first CSP bottleneck layer, the second convolution layer, the second CSP bottleneck layer, the third convolution layer, the third CSP bottleneck layer, the fourth convolution layer, and the SPP layer are cascaded in sequence; The Focus layer divides the image processed by the image preprocessing module into four parts. The depth of each part remains unchanged, and the length and width are halved. Then, the parts are spliced ​​in the depth dimension and finally convolution is performed. The first convolution layer, the second convolution layer, the third convolution layer, and the fourth convolution layer are used to perform a convolution operation of downsampling the input image by 2 times; The first CSP bottleneck layer, the second CSP bottleneck layer, and the third CSP bottleneck layer perform convolution operations on the input image; The SPP layer is used to perform the maximum pooling operation according to the maximum pooling layer with a stride of 1 and a window size of 5×5, 9×9, and 13×13, and then concatenate in the depth direction, and finally adjust the number of channels of the feature map through a 1×1 convolution.

4. The vehicle rotation detection system from the perspective of an unmanned aerial vehicle according to claim 1, characterized in that: The feature space relationship capture module includes a first Bottleneck Transformer layer and a second Bottleneck Transformer layer; The first Bottleneck Transformer layer is used to capture the spatial relationship between feature points of the feature map extracted at 32 times downsampling in the Backbone module, thereby capturing the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment; The second Bottleneck Transformer layer is used to capture the spatial relationship between feature points of the feature map extracted at 16 times downsampling in the Backbone module, and then capture the spatial relationship between vehicles and the spatial relationship between vehicles and the surrounding environment in the original image.

5. The vehicle rotation detection system from the perspective of a drone according to claim 1, characterized in that: The first feature fusion module includes: The fifth convolutional layer, the first upsampling layer, the first splicing layer, the sixth convolutional layer, the second upsampling layer, the second splicing layer, the fourth CSP bottleneck layer, the seventh convolutional layer, the third splicing layer, the fifth CSP bottleneck layer, the eighth convolutional layer, the fourth splicing layer, and the sixth CSP bottleneck layer are cascaded in sequence; The first concatenation layer is used to concatenate the feature map of the second Bottleneck Transformer layer and the feature map of the first upsampling layer; The second splicing layer is used to splice the feature map of the second CSP bottleneck layer and the feature map of the second upsampling layer; The third concatenation layer is used to concatenate the feature map of the sixth convolutional layer and the feature map of the seventh convolutional layer; The fourth concatenation layer is used to concatenate the feature map of the fifth convolution layer and the feature map of the eighth convolution layer; The fifth convolution layer and the sixth convolution layer are used to perform convolution operations on the feature map to adjust the number of feature map channels; The seventh and eighth convolutional layers are used to perform convolution operations on the feature maps with a 2x downsampling; The fourth CSP bottleneck layer extracts features from the input image and outputs a feature map with a depth of 128 and a width and height that is 8 times the downsampled size of the image obtained by the image acquisition module to the level 1 channel; The fifth CSP bottleneck layer extracts features from the input image and outputs a feature map with a depth of 256 and a width and height that is 16 times the downsampled size of the image obtained by the image acquisition module to the level 2 channel; The sixth CSP bottleneck layer extracts features from the input image and outputs a feature map with a depth of 512 and a width and height that is 32 times the downsampled size of the image acquired by the image acquisition module to the level3 channel.

6. The vehicle rotation detection system from the perspective of a drone according to claim 1, characterized in that: The second feature fusion module adjusts the three feature maps output by the first feature fusion module through constant scaling, so that the size and number of channels of the three feature maps are the same when different channels are fused. The constant scaling specifically includes: When the level 1 channel fuses the feature maps output by level 2 and level 3, the feature map output by level 2 is upsampled by 2 times, and the feature map output by level 3 is upsampled by 4 times; When the level 2 channel fuses the feature maps output by level 1 and level 3, the feature map output by level 1 is upsampled by 2 times, and the feature map output by level 3 is downsampled by 2 times using a 3×3 convolution with a stride of 2; When the level 3 channel fuses the feature maps output by level 1 and level 2, the feature map output by level 1 is first downsampled by 2 using a 3×3 convolution with a stride of 2, and then downsampled by 2 using a 3×3 maximum pooling with a stride of 2. The feature map output by level 2 is downsampled by 2 using a 3×3 convolution with a stride of 2.

7. The vehicle rotation detection system from the perspective of a drone according to claim 6, characterized in that: The second feature fusion module uses constant scaling and then performs adaptive fusion to fuse the adjusted feature maps. The adaptive fusion is expressed as: in, represents the feature map after adaptive fusion, (i, j) represents the spatial position after the fusion of the three feature maps, It represents the spatial importance parameter learned after 1×1 convolution of the feature map output by level 1 channel. It represents the spatial importance parameter learned after 1×1 convolution of the feature map output by level 2 channel. It represents the spatial importance parameter learned by the feature map of the level 3 channel output after 1×1 convolution. Represents the feature at position (i, j) in the level 1 channel, Represents the feature at position (i, j) in the level 2 channel, Represents the feature at position (i, j) in the level 3 channel.

8. The vehicle rotation detection system from the perspective of a drone according to claim 1, characterized in that: The CSL algorithm used in the CSL ring smoothing processing module is expressed as: Among them, θ represents the angle value in the current target label in the training mode, r represents the window radius, and its value is 10; g(x) represents the Gaussian function, and its expression is: Among them, μ represents the mean, which is 0; σ represents the variance, which is 10.

9. The vehicle rotation detection system from the perspective of a drone according to claim 1, characterized in that: The prediction module includes three prediction feature layers: the ninth convolutional layer, the tenth convolutional layer, the eleventh convolutional layer, the small layer, the medium layer, and the large layer, among which: The input of the ninth convolutional layer is the feature map output by the level 1 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point. The input of the tenth convolutional layer is the feature map output by the level 2 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point. The input of the eleventh convolutional layer is the feature map output by the level 3 channel. The convolution kernel size is 1×1, and the number of convolutions is determined by the number of prediction parameters for each feature point. The ninth convolutional layer, the tenth convolutional layer, and the eleventh convolutional layer have the same number of convolutions, and the number of convolutions = xywh + confidence + num_class + num_angle. Among them, xywh is the coordinate of the center point of the target on the image and the offset of the width and height of the target on the image relative to the initial anchor box, confidence refers to the confidence, num_class is the category of the target, num_angle is the category of the angle, and after using the CSL algorithm, num_angle takes an integer in the range of 0°-179°; The small layer is the feature map output by the ninth convolutional layer, which is used to predict small objects in the image; The medium layer is the feature map output by the tenth convolutional layer, which is used to predict medium-sized objects in the image; The large layer is the feature map output by the eleventh convolutional layer and is used to predict large objects in the image.

10. A vehicle rotation detection method from the perspective of an unmanned aerial vehicle, characterized in that: The following steps are involved: The image acquisition module is used to obtain images taken by the drone at an altitude of 30-80m from a direct viewpoint. Preprocessing the image in the image acquisition module through the image preprocessing module; The Backbone module is used to extract the underlying features of the image output by the image preprocessing module; The feature spatial relationship capture module is used to obtain the spatial relationship between feature points in the feature map extracted by the Backbone module at 16x downsampling and 32x downsampling, thereby capturing the spatial relationship between vehicles in the original image and the spatial relationship between vehicles and the surrounding environment. The first feature fusion module is used to fuse the features extracted from the 8-fold downsampling of the Backbone module and the features extracted from the feature space relationship capture module to obtain the fused features. The second feature fusion module is used to perform a second feature fusion on the fused features and output three feature maps of different sizes; The prediction module is used to predict the position information, angle information, and category information of the vehicle target in the image, and output the corresponding prediction confidence.

Images