Unmanned aerial vehicle aerial image target detection method based on learnable non-uniform sampling

Abstract

The invention discloses an unmanned aerial vehicle aerial image target detection method based on learnable non-uniform sampling, and belongs to the technical field of unmanned aerial vehicle aerial image target detection, and the method comprises the following steps: S1, inputting a to-be-detected image into a lightweight convolutional network, outputting a sampling point offset position, and according to the sampling point offset position, determining a to-be-detected image; a non-uniform sampling grid is obtained; s2, converting the target frame label; s3, calculating a target detection loss function by using the converted target frame label; s4, constructing a loss function based on the target amplification factor; and S5, optimizing the target detector network, and completing target detection by using the optimized target detector network. According to the method, extremely low additional calculation cost is introduced, so that the detection precision of the existing target detection algorithm for the small target in the aerial image of the unmanned aerial vehicle is remarkably improved, and the problem of small target detection of the aerial image with limited calculation resources is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection in unmanned aerial vehicle (UAV) aerial images, and in particular relates to a method for detecting targets in unmanned aerial vehicle (UAV) aerial images based on learnable non-uniform sampling. Background Art

[0002] With the development of deep convolutional neural networks (CNN) and the popularization of drone technology, drone aerial image target detection has played an important role in many remote sensing applications, such as environmental monitoring, urban management, precision agriculture and disaster response systems. Due to the large number of small-sized targets to be detected in aerial images and the uneven and sparse distribution of targets, the existing aerial image target detection algorithms mainly improve the performance of small target detection by image segmentation detection. However, such methods face problems such as high computational cost and slow reasoning speed, which may cause the target to be ignored during the image segmentation process. Therefore, how to provide a target detection method that is both computationally efficient and can significantly improve the performance of small target detection is a problem that technicians in this field need to solve urgently. Summary of the invention

[0003] In order to solve the above problems, the present invention proposes a target detection method for UAV aerial images based on learnable non-uniform sampling.

[0004] The technical solution of the present invention is: a method for detecting targets in unmanned aerial images based on learnable non-uniform sampling comprises the following steps:

[0005] S1. Input the image to be detected into the lightweight convolutional network, output the offset position of the sampling point, and obtain a non-uniform sampling grid according to the offset position of the sampling point;

[0006] S2, using a non-uniform sampling grid to obtain a non-uniformly sampled image, and converting the target box annotation;

[0007] S3, using the non-uniformly sampled image to obtain the target box prediction result, and using the converted target box annotation to calculate the target detection loss function;

[0008] S4, constructing a loss function based on the target magnification;

[0009] S5. Optimize the target detector network according to the target detection loss function and the target magnification-based loss function, and use the optimized target detector network to complete target detection.

[0010] In S5, non-uniform downsampling of the image to be detected and target detection in the distorted image space can be directly performed. Since the foreground target is explicitly magnified in the distorted image space, a better detection result for small targets can be obtained. At the same time, after obtaining the prediction result in the distorted image space, the prediction result can be mapped back to the original image space through this step to complete the target detection of the image to be detected.

[0011] The target detector network can use any convolutional neural network structure, such as the YOLO series detector or Faster-RCNN.

[0012] Furthermore, S1 includes the following sub-steps:

[0013] S11, inputting the image to be detected into a lightweight convolutional network to obtain the horizontal offset and vertical offset of each pixel coordinate;

[0014] S12, obtaining a final sampling point of each pixel coordinate according to the horizontal offset and the vertical offset of each pixel coordinate;

[0015] S13. The set of all final sampling points is used as a non-uniform sampling grid.

[0016] The beneficial effect of the above further scheme is: in the present invention, a lightweight convolutional neural network is used to construct a sampling grid matrix G for non-uniform sampling, and the process is completely differentiable, so the module can be optimized end-to-end together with the target detection network through back propagation.

[0017] In order to reduce the computational cost of this operation, it is fed into the offset estimation network f θ The original image x ori Replace it with a uniformly downsampled low-resolution image In this way, after S12-S13, the sampling grid matrix G with the same low resolution will be obtained. d , restore it to the original resolution through bilinear interpolation method, and obtain the final sampling grid matrix G. By scaling the input image size of the convolutional network, the computing time and video memory usage required in its calculation process are reduced. Since the original image size of aerial images is often large (such as 1500*2000), using low-resolution image input can significantly reduce the additional computing cost of the proposed module to maintain the high efficiency of the present invention.

[0018] Furthermore, in S12, the final sampling point (x ′ ,y ′ ) is calculated as:

[0019] x ′ =x+Δx;

[0020] y ′ =y+Δy;

[0021] Wherein, x represents the abscissa of the pixel coordinates in the image to be detected, y represents the ordinate of the pixel coordinates in the image to be detected, Δx represents the horizontal offset of the pixel coordinates, and Δy represents the vertical offset of the pixel coordinates.

[0022] Furthermore, S2 includes the following sub-steps:

[0023] S21, obtaining an image after non-uniform sampling according to a sampling grid matrix of the non-uniform sampling grid;

[0024] S22, converting the target box annotation using the image to be detected and the image after non-uniform sampling.

[0025] The beneficial effect of the above further scheme is that: in the present invention, for the target box annotation b required in the neural network training process, its upper left corner point c1 = (x1, y1) and lower right corner point c2 = (x2, y2) are mapped from the original image space to the distorted image space after non-uniform sampling to obtain c ′ 1 and c ′ 2, thus completing the distortion space target frame annotation b ′ The label mapping in the non-uniform sampling process is realized, so that the target box annotation in the distorted image space can be obtained, and the training process of the target detection network in the distorted image space can be realized.

[0026] Grid sampling method F grid_sample is the search process of the lookup table T defined by the sampling grid matrix G. The label mapping method is the reverse search process of the lookup table T By finding c1 and c2 in The nearest neighbor in space is obtained by mapping the lookup table in the distorted image space. The coordinates c on ′ 1 and c ′ 2, and finally obtain the target box annotation b in the distorted image space ′ Through a storage process of a lookup table, the non-uniform mapping of images and the conversion of labels are achieved while reducing the additional storage cost.

[0027] Furthermore, in S22, the upper left corner point and the lower right corner point of the target box annotation are mapped to the distorted image space of the image after non-uniform sampling using the original image space of the image to be detected, and the mapping expression is: Where x represents the horizontal coordinate of the pixel coordinate in the image to be detected, and y represents the vertical coordinate of the pixel coordinate in the image to be monitored. represents the distorted image space, u represents the horizontal coordinate of the pixel coordinate in the distorted image space, and v represents the vertical coordinate of the pixel coordinate in the distorted image space. Represents the original image space.

[0028] Furthermore, S3 includes the following sub-steps:

[0029] S31, inputting the non-uniformly sampled image into the target detector network to obtain the target box prediction result of the non-uniformly sampled image in the distorted image space;

[0030] S32. Calculate the target detection loss function based on the target box prediction result and the converted target box annotation.

[0031] The beneficial effect of the above further scheme is: in the present invention, the parameters of the network can be directly updated through the final target detection loss function supervision network, so that the offset estimation network can directly learn the non-uniform downsampling mode that is beneficial to improving the target detection performance.

[0032] Furthermore, in S32, the target detection loss function L Detection The expression is:

[0033]

[0034] Where α represents the first hyperparameter, x″ represents the horizontal coordinate of the center point of the target frame, y″ represents the vertical coordinate of the center point of the target frame, w represents the length of the target frame, h represents the width of the target frame, and i represents the parameter of the target frame. represents the predicted target box, b′ i represents the true target box, β represents the second hyperparameter, and p′ c represents the predicted probability of the true category of the target box, c represents the parameter of the predicted target box, represents the predicted probability of the true category of the predicted target box, and log(·) represents the logarithmic function.

[0035] The first item is the target positioning loss, which is used to measure the predicted target box b pred With the real target box b ′ The difference in space, || represents the L1 norm. The second term is the classification loss of the target, which is used to measure the predicted target box b pred Is the category of the real target box b' consistent? c and They represent the true category of the target box and the predicted probability of the predicted target box in this category. α and β are used as hyperparameters to control L Detection The weight of the localization loss and the classification loss.

[0036] Further, S4 includes the following sub-steps:

[0037] S41, calculating the target magnification according to the area size of each target frame of the image to be detected and the area size of each target frame of the image after non-uniform sampling;

[0038] S42. Construct a loss function based on the target magnification.

[0039] The beneficial effect of the above further solution is: in the present invention, the loss function L based on the supervised magnification can be Zoom , guiding the offset estimation network f θ The output can focus on the offset position of the foreground target area, thereby achieving explicit magnification of the foreground target area.

[0040] Furthermore, in S41, the target magnification m is calculated as follows:

[0041]

[0042] In the formula, a represents the area of ​​the target box of the image to be detected, a ′ represents the area of ​​the target box of the image after non-uniform sampling, and sum(·) represents the summation function.

[0043] Furthermore, in S42, the loss function L based on the target magnification is Zoom The expression is:

[0044]

[0045] Where max(·) represents the maximum function, log(·) represents the logarithmic function, α represents the first hyperparameter, β represents the second hyperparameter, γ represents the third hyperparameter, and m represents the target magnification.

[0046] The beneficial effects of the present invention are:

[0047] (1) The present invention introduces a lightweight convolutional network module before the target detector to predict the offset position of the sampling point during non-uniform sampling; based on the learned non-uniform sampling points, a target frame conversion method based on a lookup table is proposed to achieve the conversion of the target frame in the original image space to the distorted space during training, and the restoration of the target frame in the distorted image space to the original image space during prediction, thereby supporting end-to-end network optimization;

[0048] (2) The present invention proposes a loss function based on the target magnification ratio, which explicitly guides the offset network to focus on the foreground target area, thereby achieving visual magnification of the target to be detected under the same canvas size;

[0049] (3) The present invention significantly improves the detection accuracy of existing target detection algorithms for small targets in drone aerial images by introducing extremely low additional computing costs, thus solving the problem of small target detection in aerial images with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of the target detection method for UAV aerial images based on learnable non-uniform sampling. DETAILED DESCRIPTION

[0051] The embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, the present invention provides a method for detecting targets in drone aerial images based on learnable non-uniform sampling, comprising the following steps:

[0053] S1. Input the image to be detected into the lightweight convolutional network, output the offset position of the sampling point, and obtain a non-uniform sampling grid according to the offset position of the sampling point;

[0054] S2, using a non-uniform sampling grid to obtain a non-uniformly sampled image, and converting the target box annotation;

[0055] S3, using the non-uniformly sampled image to obtain the target box prediction result, and using the converted target box annotation to calculate the target detection loss function;

[0056] S4, constructing a loss function based on the target magnification;

[0057] S5. Optimize the target detector network according to the target detection loss function and the target magnification-based loss function, and use the optimized target detector network to complete target detection.

[0058] In S5, non-uniform downsampling of the image to be detected and target detection in the distorted image space can be directly performed. Since the foreground target is explicitly magnified in the distorted image space, a better detection result for small targets can be obtained. At the same time, after obtaining the prediction result in the distorted image space, the prediction result can be mapped back to the original image space through this step to complete the target detection of the image to be detected.

[0059] The target detector network can use any convolutional neural network structure, such as the YOLO series detector or Faster-RCNN.

[0060] In this embodiment of the present invention, S1 includes the following sub-steps:

[0061] S11, inputting the image to be detected into a lightweight convolutional network to obtain the horizontal offset and vertical offset of each pixel coordinate;

[0062] S12, obtaining a final sampling point of each pixel coordinate according to the horizontal offset and the vertical offset of each pixel coordinate;

[0063] S13. The set of all final sampling points is used as a non-uniform sampling grid.

[0064] In the present invention, a lightweight convolutional neural network is used to construct a sampling grid matrix G for non-uniform sampling. The process is completely differentiable, so the module can be optimized end-to-end together with the target detection network through back propagation.

[0065] In order to reduce the computational cost of this operation, it is fed into the offset estimation network f θ The original image x ori Replace it with a uniformly downsampled low-resolution image In this way, after S12-S13, the sampling grid matrix G with the same low resolution will be obtained. d , restore it to the original resolution through bilinear interpolation method, and obtain the final sampling grid matrix G. By scaling the input image size of the convolutional network, the computing time and video memory usage required in its calculation process are reduced. Since the original image size of aerial images is often large (such as 1500*2000), using low-resolution image input can significantly reduce the additional computing cost of the proposed module to maintain the high efficiency of the present invention.

[0066] In the embodiment of the present invention, in S12, the final sampling point (x ′ ,y ′ ) is calculated as:

[0067] x ′ =x+Δx;

[0068] y ′ =y+Δy;

[0069] Wherein, x represents the abscissa of the pixel coordinates in the image to be detected, y represents the ordinate of the pixel coordinates in the image to be detected, Δx represents the horizontal offset of the pixel coordinates, and Δx represents the vertical offset of the pixel coordinates.

[0070] In this embodiment of the present invention, S2 includes the following sub-steps:

[0071] S21, obtaining an image after non-uniform sampling according to a sampling grid matrix of the non-uniform sampling grid;

[0072] S22, converting the target box annotation using the image to be detected and the image after non-uniform sampling.

[0073] In the present invention, for the target box annotation b required in the neural network training process, its upper left corner point c1 = (x1, y1) and lower right corner point c2 = (x2, y2) are mapped from the original image space to the distorted image space after non-uniform sampling to obtain c ′ 1 and c ′ 2, thus completing the distortion space target frame annotation b ′ The label mapping in the non-uniform sampling process is realized, so that the target box annotation in the distorted image space can be obtained, and the training process of the target detection network in the distorted image space can be realized.

[0074] Grid sampling method F grid_sample is the search process of the lookup table T defined by the sampling grid matrix G. The label mapping method is the reverse search process of the lookup table T By finding c1 and c2 in The nearest neighbor in space is obtained by mapping the lookup table in the distorted image space. The coordinates c on ′ 1 and c ′ 2, and finally obtain the target box annotation b in the distorted image space ′ Through a storage process of a lookup table, the non-uniform mapping of images and the conversion of labels are achieved while reducing the additional storage cost.

[0075] In the embodiment of the present invention, in S22, the upper left corner point and the lower right corner point of the target box annotation are mapped to the distorted image space of the image after non-uniform sampling using the original image space of the image to be detected, and the mapping expression is: Where x represents the horizontal coordinate of the pixel coordinate in the image to be detected, and y represents the vertical coordinate of the pixel coordinate in the image to be monitored. represents the distorted image space, u represents the horizontal coordinate of the pixel coordinate in the distorted image space, and v represents the vertical coordinate of the pixel coordinate in the distorted image space. Represents the original image space.

[0076] In this embodiment of the present invention, S3 includes the following sub-steps:

[0077] S31, inputting the non-uniformly sampled image into the target detector network to obtain the target box prediction result of the non-uniformly sampled image in the distorted image space;

[0078] S32. Calculate the target detection loss function based on the target box prediction result and the converted target box annotation.

[0079] In the present invention, the parameters of the network can be directly updated through the final target detection loss function supervision network, so that the offset estimation network can directly learn the non-uniform downsampling mode that is beneficial to improving the target detection performance.

[0080] In the embodiment of the present invention, in S32, the target detection loss function L Detection The expression is:

[0081]

[0082] Where α represents the first hyperparameter, x″ represents the horizontal coordinate of the center point of the target frame, y″ represents the vertical coordinate of the center point of the target frame, w represents the length of the target frame, h represents the width of the target frame, and i represents the parameter of the target frame. represents the predicted target box, b ′ i represents the true target box, β represents the second hyperparameter, and p ′ c represents the predicted probability of the true category of the target box, c represents the parameter of the predicted target box, represents the predicted probability of the true category of the predicted target box, and log(·) represents the logarithmic function.

[0083] The first item is the target positioning loss, which is used to measure the predicted target box b pred The difference between the real target box b′ and the real target box b′ in space, || represents the L1 norm. The second term is the classification loss of the target, which is used to measure the predicted target box b pred Is the category of the real target box b' consistent? c and They represent the true category of the target box and the predicted probability of the predicted target box in this category. α and β are used as hyperparameters to control L Detection The weight of the localization loss and the classification loss.

[0084] In this embodiment of the present invention, S4 includes the following sub-steps:

[0085] S41, calculating the target magnification according to the area size of each target frame of the image to be detected and the area size of each target frame of the image after non-uniform sampling;

[0086] S42. Construct a loss function based on the target magnification.

[0087] The beneficial effect of the above further solution is: in the present invention, the loss function L based on the supervised magnification can be Zoom , guiding the offset estimation network f θ The output can focus on the offset position of the foreground target area, thereby achieving explicit magnification of the foreground target area.

[0088] In the embodiment of the present invention, in S41, the calculation formula of the target magnification m is:

[0089]

[0090] In the formula, a represents the area of ​​the target box of the image to be detected, a ′ represents the area of ​​the target box of the image after non-uniform sampling, and sum(·) represents the summation function.

[0091] In the embodiment of the present invention, in S42, the loss function L based on the target magnification Zoom The expression is:

[0092]

[0093] Where max(·) represents the maximum function, log(·) represents the logarithmic function, α represents the first hyperparameter, β represents the second hyperparameter, γ represents the third hyperparameter, and m represents the target magnification.

[0094] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for detecting targets in drone aerial images based on learnable non-uniform sampling, characterized in that: The following steps are involved: S1. Input the image to be detected into the lightweight convolutional network, output the offset position of the sampling point, and obtain a non-uniform sampling grid according to the offset position of the sampling point; S2, using a non-uniform sampling grid to obtain a non-uniformly sampled image, and converting the target box annotation; S3, using the non-uniformly sampled image to obtain the target box prediction result, and using the converted target box annotation to calculate the target detection loss function; S4, constructing a loss function based on the target magnification; S5. Optimize the target detector network according to the target detection loss function and the target magnification-based loss function, and use the optimized target detector network to complete target detection.

2. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 1, characterized in that: The S1 comprises the following sub-steps: S11, inputting the image to be detected into a lightweight convolutional network to obtain the horizontal offset and vertical offset of each pixel coordinate; S12, obtaining a final sampling point of each pixel coordinate according to the horizontal offset and the vertical offset of each pixel coordinate; S13. The set of all final sampling points is used as a non-uniform sampling grid.

3. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 2, characterized in that: In S12, the final sampling point (x ′ ,y ′ ) is calculated as: x ′ =x+Δx; y ′ =y+Δy; Wherein, x represents the abscissa of the pixel coordinates in the image to be detected, y represents the ordinate of the pixel coordinates in the image to be detected, Δx represents the horizontal offset of the pixel coordinates, and Δx represents the vertical offset of the pixel coordinates.

4. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 1, characterized in that: The S2 comprises the following sub-steps: S21, obtaining an image after non-uniform sampling according to a sampling grid matrix of the non-uniform sampling grid; S22, converting the target box annotation using the image to be detected and the image after non-uniform sampling.

5. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 4, characterized in that: In S22, the upper left corner point and the lower right corner point of the target box annotation are mapped to the distorted image space of the image after non-uniform sampling using the original image space of the image to be detected, and the mapping expression is: Where x represents the horizontal coordinate of the pixel coordinate in the image to be detected, and y represents the vertical coordinate of the pixel coordinate in the image to be monitored. represents the distorted image space, u represents the horizontal coordinate of the pixel coordinate in the distorted image space, and v represents the vertical coordinate of the pixel coordinate in the distorted image space. Represents the original image space.

6. The method for detecting targets in drone aerial images based on learnable non-uniform sampling according to claim 1, characterized in that: The S3 comprises the following sub-steps: S31, inputting the non-uniformly sampled image into the target detector network to obtain the target box prediction result of the non-uniformly sampled image in the distorted image space; S32. Calculate the target detection loss function based on the target box prediction result and the converted target box annotation.

7. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 6, characterized in that: In S32, the target detection loss function L Detection The expression is: Where α represents the first hyperparameter, x″ represents the horizontal coordinate of the center point of the target frame, y″ represents the vertical coordinate of the center point of the target frame, w represents the length of the target frame, h represents the width of the target frame, and i represents the parameter of the target frame. represents the predicted target box, b ′ i represents the true target box, β represents the second hyperparameter, and p ′ c represents the predicted probability of the true category of the target box, c represents the parameter of the predicted target box, represents the predicted probability of the true category of the predicted target box, and log(·) represents the logarithmic function.

8. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 1, characterized in that: The S4 comprises the following sub-steps: S41, calculating the target magnification according to the area size of each target frame of the image to be detected and the area size of each target frame of the image after non-uniform sampling; S42. Construct a loss function based on the target magnification.

9. The method for detecting targets in unmanned aerial images based on learnable non-uniform sampling according to claim 8, characterized in that: In S41, the target magnification m is calculated as follows: In the formula, a represents the area of ​​the target box of the image to be detected, a ′ represents the area of ​​the target box of the image after non-uniform sampling, and sum(·) represents the summation function.

10. The method for detecting targets in drone aerial images based on learnable non-uniform sampling according to claim 8, characterized in that: In S42, the loss function L based on the target magnification Zoom The expression is: Where max(·) represents the maximum function, log(·) represents the logarithmic function, α represents the first hyperparameter, β represents the second hyperparameter, γ represents the third hyperparameter, and m represents the target magnification.

Images