A structure-guided super-resolution network construction method for rotary-wing UAV images

Abstract

The present invention discloses a method for constructing a structure-guided rotorcraft image super-resolution network, belonging to the field of UAV super-resolution. The method constructs a structure-guided rotorcraft super-resolution network, performs gradient calculation on the acquired low-resolution image to obtain a low-resolution gradient map, fuses the two at the channel level, and then extracts features to obtain a feature map. The low-resolution key point heat map is predicted based on the feature map, and the feature map is then upsampled and processed separately to obtain a high-resolution UAV gradient map and a high-resolution UAV image. Loss calculation is performed on the high-resolution gradient map, the low-resolution key point heat map, and the true label results. A discriminator is used to determine the probability that the output image is a true image to enhance the reconstruction capability of the super-resolution network, forming an end-to-end constraint. The network can output super-resolution images of rotorcraft with excellent visual quality, improving the detection and monitoring effects of long-range rotorcraft imaging.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) super-resolution, and more specifically, relates to a structure-guided rotorcraft UAV image super-resolution network construction method. Background Art

[0002] Rotary-wing drones (UAVs) are becoming increasingly common flying devices due to their low price, ease of control, high maneuverability, and wide field of view. More and more consumers are flying them in urban areas and natural scenic areas where no flight bans are in place. The illegal operation of UAVs poses significant challenges to public security and airport operations. Therefore, the detection and identification of UAVs is crucial. Because UAVs are generally small, have low power, and are considered weak targets, long-range detection often results in the target being too small and having low resolution. This poses challenges to long-range UAV monitoring and detection.

[0003] Super-resolution (SR) involves reconstructing a corresponding high-resolution image from an observed low-resolution image using software or hardware methods. The reconstructed image contains the same scene, object, and semantic information as the original image. Super-resolution technology has important applications in surveillance equipment, satellite imagery, remote sensing, digital high-definition imaging, microscopy, video coding communications, video restoration, and medical imaging. Super-resolution can provide a clearer view of drone structures from low-resolution drone image signals, improving the system's ability to monitor and counter rotary-wing drones.

[0004] Currently, there are no super-resolution methods for rotary-wing drones. Patent CN113379597A, based on the SRFBN network, introduces prior information on facial key points and an attention mechanism to design a facial super-resolution reconstruction network. This method feeds the key point heat map back to the super-resolution module and achieves super-resolution through a loop iteration. However, due to the significant computational overhead required for key point detection and loop iteration during the inference phase, this method has low operational efficiency and is difficult to implement in terminal deployment applications. Patent CN114463174A proposes an image super-resolution method based on meta-learning and structure preservation. During inference, this method uses a network to process the original image and gradient map separately, ultimately obtaining a high-resolution image by fusing the image and gradient information. Because this method uses different network structures for processing the original image and gradient map, it also incurs a significant computational overhead. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a structure-guided rotorcraft UAV image super-resolution network construction method, which fully utilizes the UAV gradient and key point information to reconstruct an accurate and clear UAV structure, while reducing the computational overhead during network inference and improving operational efficiency.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a structure-guided method for constructing a super-resolution network for rotary-wing UAV images is provided, comprising:

[0007] A structure-guided rotorcraft UAV image super-resolution network is constructed, comprising: a preprocessing module for preprocessing the UAV low-resolution image, wherein the preprocessing includes image normalization; a gradient calculation module for obtaining a gradient map of the preprocessed UAV low-resolution image; a fusion module for performing channel-level fusion on the preprocessed low-resolution image and the gradient map to obtain a fused image; a feature extraction network for extracting features of the fused image to obtain a low-resolution feature map; a super-resolution module for upsampling the low-resolution feature map to obtain a high-resolution feature map; a key point prediction module, a high-resolution gradient map prediction module, and a high-resolution image prediction module, respectively, for predicting various key point heat maps, high-resolution gradient maps, and high-resolution images of the UAV based on the high-resolution feature map; and a discriminator for calculating the probability that the high-resolution image generated by the high-resolution image prediction module is a true image when training the network.

[0008] The super-resolution network is supervisedly trained using low-resolution images of the drone as samples and various key point heat maps, high-resolution gradient maps, and high-resolution images corresponding to the low-resolution images of the drone as labels to obtain a trained super-resolution network.

[0009] According to a second aspect of the present invention, a structure-guided method for constructing a super-resolution network for rotary-wing UAV images is provided, comprising:

[0010] The low-resolution image of the drone to be processed is input into the super-resolution network constructed by the method described in the first aspect to obtain a high-resolution image of the low-resolution image of the drone to be processed.

[0011] According to a third aspect of the present invention, there is provided an electronic device comprising: a computer-readable storage medium and a processor;

[0012] The computer-readable storage medium is used to store executable instructions;

[0013] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to the first aspect or the method according to the second aspect.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method as described in the first aspect or the method as described in the second aspect.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0016] 1. The method provided by the present invention simultaneously utilizes gradients and keypoints, two features that can characterize drone structure, to guide super-resolution of rotary-wing drones, thereby improving the accuracy of super-resolution reconstruction. Furthermore, a multi-task learning network is constructed, in which high-resolution gradient map prediction, keypoint prediction, and high-resolution drone prediction are each performed using lightweight branches. During inference, only the high-resolution drone prediction branch is used to reconstruct a clear drone structure, saving significant computational overhead and achieving high efficiency. The multi-task learning network is a structure-guided rotary-wing drone super-resolution network. This network performs channel-machine fusion on a low-resolution image and its low-resolution gradient map, which is then processed by a feature extraction network to obtain a feature map. The feature map is input into the network's keypoint prediction module, which outputs a low-resolution keypoint heatmap. This feature map is then input into a super-resolution processing module to further obtain a high-resolution drone gradient map and a high-resolution drone image. Loss calculation and supervision constraints are performed on the high-resolution gradient map, the low-resolution keypoint heatmap, and the true labels. A discriminator determines the probability that the output image is a true image to enhance the reconstruction capability of the super-resolution network. The network is trained until convergence. The network can output super-resolution images of rotary-wing UAVs with good visual effects, improving the detection and monitoring effects of long-range rotary-wing UAV imaging.

[0017] 2. The method provided by the present invention, when constructing the training dataset for the above-mentioned multi-task learning network, performs image enhancement and screening on the high-resolution drone original image, and then crops and resizes it according to the drone position and annotation information, and then performs corresponding processing to obtain low-resolution sample images of the drone and their corresponding key point heat map labels, high-resolution gradient map labels, and high-resolution image labels. This can further improve the visibility of the target and the retention of texture details after the high-resolution image is downsampling, ensuring the basis for super-resolution reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the overall process of the structure-guided super-resolution network construction method for a rotary-wing UAV provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the training process of a super-resolution network for a rotary-wing UAV provided in an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the inference process of a super-resolution network for a rotary-wing UAV provided in an embodiment of the present invention;

[0021] Figure 4 This is a structural diagram of the super-resolution network of a rotary-wing UAV provided by an embodiment of the present invention;

[0022] Figure 5 This is an example diagram of a feature extraction network in a super-resolution network for a rotary-wing UAV provided by an embodiment of the present invention;

[0023] Figure 6 Schematic diagram of the residual network used in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0025] Existing super-resolution methods, in addition to incurring significant computational overhead, only utilize keypoint information and gradient information, respectively. While using only keypoint information can focus on the local features of the target image to a certain extent, it tends to ignore the image's overall structural information, resulting in distortion or blurred key areas in the generated high-resolution image. On the other hand, using only gradient information may cause the model framework to focus more on reconstructing edges and details, neglecting the precise restoration of key areas, resulting in distortion of key features or discrepancies with the true image. This processing approach typically fails to simultaneously account for both the global structure and detailed information of the image, limiting the quality of super-resolution reconstruction.

[0026] Based on this, an embodiment of the present invention provides a structure-guided rotorcraft UAV image super-resolution network construction method, comprising:

[0027] A structure-guided rotorcraft UAV image super-resolution network is constructed, comprising: a preprocessing module for preprocessing the UAV low-resolution image, wherein the preprocessing includes image normalization; a gradient calculation module for obtaining a gradient map of the preprocessed UAV low-resolution image; a fusion module for performing channel-level fusion on the preprocessed low-resolution image and the gradient map to obtain a fused image; a feature extraction network for extracting features of the fused image to obtain a low-resolution feature map; a super-resolution module for upsampling the low-resolution feature map to obtain a high-resolution feature map; a key point prediction module, a high-resolution gradient map prediction module and a high-resolution image prediction module, respectively for predicting various key point heat maps, high-resolution gradient maps and high-resolution images of the UAV based on the high-resolution feature map; and a discriminator for calculating the probability that the high-resolution image generated by the high-resolution image prediction module is a real image when training the network.

[0028] The super-resolution network is supervisedly trained using low-resolution images of the drone as samples and various key point heat maps, high-resolution gradient maps, and high-resolution images corresponding to the low-resolution images of the drone as labels to obtain a trained super-resolution network.

[0029] Specifically, after constructing the structure-guided rotorcraft super-resolution network, it is necessary to construct a data set to train the structure-guided rotorcraft super-resolution network. The sample images in the data set are low-resolution images of the drone, and the label images are various key point heat maps, high-resolution gradient maps and high-resolution images corresponding to the sample images.

[0030] It can be understood that the corresponding low-resolution image of the drone can be obtained by downsampling the high-resolution rotorcraft original image, and the corresponding various key point heat maps and high-resolution gradients can be obtained respectively through heat map calculation and gradient calculation, and the high-resolution rotorcraft original image can be used as the high-resolution image.

[0031] In order to further improve the visibility of the target and the preservation of texture details after downsampling the high-resolution image, and to ensure the basis for super-resolution reconstruction, as a further preferred embodiment of the present invention, the acquisition method of the drone low-resolution image and its corresponding various key point heat maps, high-resolution gradient maps and high-resolution images is as follows:

[0032] After performing image enhancement on the captured high-resolution drone original image, images with drone target detection frame pixels exceeding a pixel threshold are selected as the first target image. The drone's position coordinates and key point coordinates in the first target image are annotated. Based on the annotated information, the first target image is cropped to leave only the drone area to obtain a second target image. The second target image is resized to meet the required size and then downsampled to obtain the low-resolution drone image.

[0033] Generating a heat map of each key point based on the coordinates of each key point in the first target image, superimposing the heat maps of key points of the same category, and then cropping and transforming each type of superimposed heat map according to the size of the second target image to obtain a heat map of each key point corresponding to the low-resolution image of the drone;

[0034] The second target image is a high-resolution image corresponding to the low-resolution image of the drone; and the gradient map of the high-resolution image is a high-resolution gradient map corresponding to the low-resolution image of the drone.

[0035] Specifically, first, a rotary-wing drone original image is captured. The high-resolution drone original image is an RGB image or a grayscale image. For example, if the high-resolution drone original image is an RGB image, then three-channel color rotary-wing drone images are captured under different lighting conditions. This embodiment of the present invention is described using the high-resolution drone original image as an RGB image as an example.

[0036] After collecting drone images, image enhancement is required. Image enhancement operations include rotation, translation, cropping, flipping, and adding noise. These operations are performed randomly to generate more diverse variations, making the data closer to real-world data and improving data coverage.

[0037] Then, the images that meet the preset conditions in the enhanced image of the rotorcraft UAV are selected for UAV position and key point calibration. The preset condition is that the pixels of the UAV target detection frame exceed the pixel threshold. The pixel threshold can be set according to the actual situation, for example: 200×200.

[0038] After obtaining the specific position of the rotary-wing UAV in the image and marking it, key points of various parts of the rotary-wing UAV are marked.

[0039] It can be understood that the key points of the rotor UAV are the center points of the top of each rotor of the UAV and the center point of the battery part of the rotor UAV fuselage.

[0040] Taking a quad-rotor drone as an example, the key points of each part of the quad-rotor drone include: the center point of the top of the four rotors and the center point of the battery part of the quad-rotor drone body.

[0041] The location and key point annotation information of the quadrotor can be expressed in the following format: [({'bbox':array([x_1,y_1,x_2,y_2],dtype=int64)}),(img_name,{'landmark':array([[x1,y1],[x2,y2],[x3,y3],[x4,y4],[x5,y5]])})]. Here, x_1 and y_1 represent the pixel coordinates of the upper left corner of the rotorcraft, and x_2 and y_2 represent the pixel coordinates of the lower right corner of the rotorcraft. x1, x2, x3, and x4 are the horizontal coordinates of the rotor tip of the quadrotor, and y1, y2, y3, and y4 are the vertical coordinates of the rotor tip of the quadrotor. x5 is the horizontal coordinate of the battery part of the quadrotor, and y5 is the vertical coordinate of the battery part of the quadrotor.

[0042] According to the position of the rotary-wing UAV in the image and the corresponding key point annotation information, the pixel coordinates of each key point in the corresponding rotary-wing UAV image are obtained, and a heat map of each key point is constructed as follows:

[0043]

[0044] In this embodiment, the Gaussian peak parameters are set as follows: the standard deviation σ in the xy direction is 25, the maximum value of the Gaussian peak is 255, (x, y) is the coordinate of the key point, and (i, j) is the coordinate of each pixel in the key point heat map.

[0045] Taking a quadcopter as an example, since a quadcopter has four rotors corresponding to four key points, and one key point corresponds to one heat map, multiple Gaussian peaks are superimposed on a heat map in the heat map of such key points, that is, the heat maps of the center points of all rotor tips are superimposed into one heat map.

[0046] The following processing is performed on the pictures that meet the preset conditions in the original image of the rotary-wing drone to obtain a three-channel RGB image of the low-resolution rotary-wing drone that meets the input requirements of the super-resolution network, specifically including:

[0047] The high-resolution drone image (i.e., the image that meets the preset conditions in the original image of the rotary-wing drone) is cropped. Based on the drone position annotation information, the background in the image is cropped, leaving only the drone area. Then, the cropped high-resolution drone image is resized to a uniform size (the specific uniform size can be set according to actual requirements. In the embodiment of the present invention, the uniform size is 256×256 as an example), and then downsampled to a three-channel RGB color image of a specific width and length to obtain a low-resolution image. The key point information and key point heat map annotated in the obtained low-resolution rotary-wing drone image are transformed accordingly to obtain a low-resolution drone image and related key point information. The transformation formula is:

[0048]

[0049] Among them, x i is the horizontal coordinate of the point in the cropping area at the original size, x i ′ is the horizontal axis coordinate after cropping and downsampling, k x is the horizontal axis downsampling ratio (k x >1), The starting horizontal coordinate of the image cropping, scale x is the scale factor, which is equal to the image width after resize (i.e., size transformation) divided by the image width after crop (i.e., cropping) multiplied by the inverse of the sampling rate. i ′ The calculation of y is similar, i is the vertical coordinate of the point in the cropping area at the original size, y i ′ is the vertical axis coordinate after cropping and downsampling, k y is the vertical axis downsampling ratio (k y >1), The starting vertical coordinate for image cropping.

[0050] As an example, in this embodiment, the resize value is set to 256×256, and the downsampling ratio is set to 4 times.

[0051] like Figure 4 As shown, the structure-guided rotorcraft super-resolution network includes a gradient calculation module, a fusion module, a feature extraction network, a super-resolution module, a key point prediction module, a high-resolution gradient map prediction module, a high-resolution image prediction module and a discriminator.

[0052] The following describes the various modules of the network in conjunction with the training process of the structure-guided rotorcraft super-resolution network. The training process of the structure-guided rotorcraft super-resolution network includes the following steps:

[0053] (1) Before inputting the low-resolution image data from the drone into the network, it needs to be preprocessed to standardize the data and adapt it to the input requirements of the network model. The preprocessing operation includes image normalization to adjust the image pixels to a fixed range to facilitate the subsequent calculation of the network; further, the preprocessing operation also includes color space conversion and / or denoising. Among them, when the network model processes visible light data and infrared data simultaneously for training and testing, color space conversion can be performed to convert the image color space into RGB format so that the infrared single-channel image can be processed as a color image.

[0054] (2) The gradient calculation module processes the low-resolution image to obtain a gradient map.

[0055] The Sobel operator is usually used to extract gradients in both the x and y directions. When extracting gradients from a multi-channel image (such as an RGB image), the RGB image must first be converted to a single-channel grayscale image. This conversion combines color information into a single brightness value using the following conversion formula:

[0056] I=0.2989·R+0.5870·G+0.1140·B (4)

[0057] The following calculations are then required:

[0058] I X (x)=I(x+1,y)-I(x-1,y), (5)

[0059]

[0060] Here, the coordinate (x, y) is the position of a pixel in the low-resolution image, I(x, y) is the pixel value, and M(·) represents the operation of extracting the gradient map, whose elements are the gradient lengths of the pixel at coordinate x = (x, y). This operation can be easily implemented using a convolutional layer with a fixed convolution kernel.

[0061] (3) The fusion module performs channel-level fusion on the low-resolution UAV image and gradient map and outputs the fused data.

[0062] When performing image channel-level fusion, the input low-resolution drone image and gradient map Perform channel connection to obtain fused data Where M is the image width, N is the image length, and c is the number of channels in the input low-resolution drone image. Specifically, if the high-resolution drone original image is an RGB image or a grayscale image, c is set to 3 and 1, respectively.

[0063] (4) After the channel-level fusion data of the above-mentioned low-resolution UAV image and low-resolution gradient map are input into the feature extraction network, a low-resolution UAV feature map is output.

[0064] The feature extraction network uses an improved HRNet network. Based on the HRNet network, the input dimension of the first layer is set to c+1, the feature dimension of the network is set to 256, the low-resolution module in the last layer of feature extraction module is removed, and only the highest-resolution module is retained to output the drone feature map. Where M is the image width, N is the image length, and D is the feature dimension, which can be set according to actual conditions. For example, it can be set to 256. The network structure of the feature extraction module is as follows: Figure 5 shown.

[0065] (5) The low-resolution UAV feature map is input into the key point prediction module to predict the key points of the UAV. The output form of the key points is a heat map.

[0066] The key point prediction module consists of a residual module and a 1×1 convolution. The input and output dimensions of the residual module and the input dimension of the 1×1 convolution are both 256. The output dimension of the 1×1 convolution is the number of key point categories. Usually, the key point categories are the rotor center point and the drone battery center point, that is, the number of key point categories is 2.

[0067] Heatmap is a commonly used keypoint representation method used to indicate the location of keypoints and their confidence. Heatmaps are usually generated by mapping the locations of keypoints onto a two-dimensional grid and using a Gaussian distribution around the keypoint locations. The formula is as follows:

[0068]

[0069] During network training, the coordinates of the key points returned after regression tend to overfit on the training set. Therefore, using this method, a heat map with the number of channels equal to the number of key point categories can be returned to train a robust network.

[0070] (6) The low-resolution UAV feature map is input into the super-resolution module, which outputs the UAV high-resolution feature map.

[0071] The super-resolution module consists of two upsampling modules, each of which consists of a residual module and a PixelShuffle layer. The input and output dimensions of the residual module and the PixelShuffle layer are both 256.

[0072] Among them, the specific settings of the residual module are: a convolution kernel of 1×1, the number of input channels is 256, and the number of output channels is 128; a batch normalization layer with an input channel number of 128; a ReLU activation layer. A convolution layer with a convolution kernel size of 3×3, the number of input channels is 128, the number of output channels is 128, padding is 1, and bias is set to True; a batch normalization layer with an input channel number of 128, and a ReLU activation layer. A convolution layer with a convolution kernel size of 1*1, the number of input channels is 128, the number of output channels is 256, and bias is set to True. In addition, the final output tensor needs to be superimposed with the initial input tensor. See for details. Figure 6 .

[0073] (7) The high-resolution feature map of the UAV is input into the high-resolution gradient map prediction module, which outputs the high-resolution UAV gradient map.

[0074] The high-resolution gradient map prediction module consists of the above residual module and a 1×1 convolution. The input and output dimensions of the residual module and the input dimension of the 1×1 convolution are both 256, and the output dimension of the 1×1 convolution is 1.

[0075] (8) The high-resolution feature map of the UAV is input into the high-resolution image prediction module, which outputs the high-resolution UAV image.

[0076] The high-resolution image prediction module consists of the aforementioned residual module and a 1×1 convolution. The input and output dimensions of the residual module and the input dimension of the 1×1 convolution are both 256, and the output dimension of the 1×1 convolution is 3. It is understandable that when the original high-resolution drone image is a grayscale image, the output dimension of the 1×1 convolution is 1.

[0077] (9) Compare the gradient map and key point heat map of the real high-resolution drone image with the high-resolution drone gradient map obtained in step (7) and the key point heat map output in step (5), and calculate the loss. The gradient map reflects the structure and contour edges of the drone, while the key points refer to the fine-grained representation of the overall structure of the drone. These two serve as drone structural constraints and contribute to the super-resolution reconstruction of drone images.

[0078] Loss calculation includes: obtaining the heat map label G of each key point g , get high-resolution drone label H g , get the high-resolution gradient map label G g ; Calculate the loss according to the following formula:

[0079] loss = λ g ‖G g -G p ‖+λ h ‖Hg -H p ‖+λ heat ‖Heat g -Heat p ‖+λ adv l d , (10)

[0080] Among them, λ g ,λ h ,λ heat ,λ adv are the weight parameters of heat map, high-resolution UAV, high-resolution gradient map and discriminator loss respectively, ‖·‖ is the F-1 norm; l d is the discriminator loss, calculated by cross entropy loss, and the calculation formula is as follows:

[0081] L D =-Log(D(I HR ))-Log(1-D(G(I LR ))), (11)

[0082] Among them, D(I HR ) represents the predicted probability of the discriminator for the real image, D(G(I LR )) represents the predicted probability of the discriminator for the generated image.

[0083] As an example, the initial value of the learning rate of this network is set to 0.01, and it is multiplied by 0.1 every 20 epochs; the optimizer uses ADAM, and the parameters are set to β1 = 0.9, β2 = 0.999, and ε = 1×10 -8 ; The hyperparameter for loss calculation is set to λ g =λ h =λ heat =1,λ adv =0.001, and the number of training rounds is 100.

[0084] (10) The generated high-resolution drone image and the real high-resolution drone image are input into the discriminator, and the output image is the probability that the image is a real image. This module can promote the rotorcraft drone super-resolution network to generate more realistic high-resolution images.

[0085] It can be understood that after the network is trained, the network in inference mode inputs low-resolution rotorcraft images, and the output is super-resolution drone images.

[0086] An embodiment of the present invention provides a structure-guided method for constructing a super-resolution network for rotary-wing UAV images, comprising:

[0087] The low-resolution image of the drone to be processed is input into the super-resolution network constructed by the method described in any of the above embodiments to obtain a high-resolution image of the low-resolution image of the drone to be processed.

[0088] An embodiment of the present invention provides an electronic device, characterized by comprising: a computer-readable storage medium and a processor;

[0089] The computer-readable storage medium is used to store executable instructions;

[0090] The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method described in any one of the above embodiments.

[0091] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0092] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A structure-guided super-resolution network construction method for rotorcraft UAV images, characterized by: include: A structure-guided rotorcraft UAV image super-resolution network is constructed, comprising: a preprocessing module for preprocessing the UAV low-resolution image, wherein the preprocessing includes image normalization; a gradient calculation module for obtaining a gradient map of the preprocessed UAV low-resolution image; a fusion module for performing channel-level fusion on the preprocessed low-resolution image and the gradient map to obtain a fused image; a feature extraction network for extracting features of the fused image to obtain a low-resolution feature map; a super-resolution module for upsampling the low-resolution feature map to obtain a high-resolution feature map; a key point prediction module, a high-resolution gradient map prediction module, and a high-resolution image prediction module, respectively, for predicting various key point heat maps, high-resolution gradient maps, and high-resolution images of the UAV based on the high-resolution feature map; and a discriminator for calculating the probability that the high-resolution image generated by the high-resolution image prediction module is a true image when training the network. The super-resolution network is supervisedly trained using low-resolution images of the drone as samples and various key point heat maps, high-resolution gradient maps, and high-resolution images corresponding to the low-resolution images of the drone as labels to obtain a trained super-resolution network.

2. The method according to claim 1, wherein The pre-processing further includes color space conversion and / or denoising.

3. The method according to claim 1 or 2, wherein: The method for obtaining the low-resolution image of the drone and the corresponding key point heat maps, high-resolution gradient maps and high-resolution images is as follows: After image enhancement is performed on the high-resolution drone original image, images whose drone target detection frame pixels exceed the pixel threshold are selected as the first target image. The position coordinates and key point coordinates of the drone in the first target image are annotated. The first target image is cropped based on the position annotation information to leave only the drone area to obtain the second target image. The second target image is resized to meet the required size and then downsampled to obtain the low-resolution drone image. Generating a heat map of each key point based on the coordinates of each key point in the first target image, superimposing the heat maps of key points of the same category, and then cropping and transforming each type of superimposed heat map according to the size of the second target image to obtain a heat map of each key point corresponding to the low-resolution image of the drone; The second target image is a high-resolution image corresponding to the low-resolution image of the drone; and the gradient map of the high-resolution image is a high-resolution gradient map corresponding to the low-resolution image of the drone.

4. The method according to claim 3, wherein The high-resolution drone original image includes an RGB image or a grayscale image.

5. The method according to claim 1, wherein The loss function of the super-resolution network is: in, 、 、 、 are the weight parameters of heat map, high-resolution drone, high-resolution gradient map and discriminator loss respectively, is the F-1 norm; is the discriminator loss.

6. The method according to claim 1, wherein The feature extraction network is obtained by removing the low-resolution module in the last layer of the feature extraction module of the HRNet network.

7. The method according to claim 1, wherein The key point prediction module includes a residual module and a convolutional layer whose output dimension is the number of key point categories connected in sequence; The high-resolution gradient map prediction module includes a residual module and a convolutional layer with an output dimension of 1 connected in sequence; The high-resolution image prediction module includes a residual module and a convolutional layer with an output dimension of K connected in sequence; where K is the number of channels of the high-resolution drone original image.

8. A structure-guided super-resolution network construction method for rotorcraft UAV images, characterized in that: include: The low-resolution image of the drone to be processed is input into the super-resolution network constructed by the method described in any one of claims 1 to 7 to obtain a high-resolution image of the low-resolution image of the drone to be processed.

9. An electronic device, characterized in that: include: Computer-readable storage media and processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 7 or the method according to claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 7 or the method according to claim 8.

Images