Structure-guided rotor unmanned aerial vehicle image super-resolution network construction method

Through the structure-guided rotor drone image super-resolution network, multi-task learning is used to utilize gradient and key point information, solving the problem of large computing overhead in the existing technology, and achieving efficient rotor drone super-resolution reconstruction and Improved detection effect.

CN118967452AActive Publication Date: 2024-11-15HUAZHONG UNIV OF SCI & TECH

Patent Information

Application Number
CN202411184254.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-15
Estimated Expiration
2044-08-27

Smart Images

  • Figure CN118967452A_ABST
    Figure CN118967452A_ABST
Patent Text Reader

Abstract

The invention discloses a structure-guided rotor unmanned aerial vehicle image super-resolution network construction method, and belongs to the field of unmanned aerial vehicle super-resolution, and the method comprises the steps: constructing a structure-guided rotor unmanned aerial vehicle super-resolution network, carrying out the gradient calculation of an obtained low-resolution image, and obtaining a low-resolution gradient map; performing channel-level fusion on the two images, performing feature extraction to obtain a feature map, predicting a low-resolution key point thermodynamic diagram according to the feature map, performing up-sampling on the feature map, and performing processing to obtain a high-resolution unmanned aerial vehicle gradient map and a high-resolution unmanned aerial vehicle image; loss calculation is carried out on the results of the high-resolution gradient map, the low-resolution key point thermodynamic diagram and the real label, a discriminator is adopted to discriminate the probability that an output image is a real image so as to enhance the reconstruction capability of the super-resolution network, and end-to-end constraints are formed. The network can output a rotor unmanned aerial vehicle super-resolution image with a good visual effect, and the detection and monitoring effects of long-distance rotor unmanned aerial vehicle imaging are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) super-resolution, and more specifically, relates to a structure-guided rotorcraft UAV image super-resolution network construction method. Background Art

[0002] Rotary-wing drones have become increasingly common flying props due to their low price, easy control, high flexibility, and wide field of view. However, not only are more and more consumers flying rotary-wing drones in urban spaces and natural scenic spots without a no-fly order, they are also increasingly being used in military applications as reconnaissance or attack weapons. The illegal flying of rotary-wing drones has brought huge challenges to public security, airport operations, military control, etc., and rotary-wing drones can also carry weapons and ammunition, which poses a high threat to personnel and vehicles and other facilities. Therefore, the detection and identification of rotary-wing drones is of great significance. In today's military field, the demand for means to detect illegal drones in no-fly time and space is becoming increasingly strong. Since rotary-wing drones are generally small in size, weak in energy, and belong to weak targets, when conducting long-distance detection, the drone target size is often too small and the resolution is low, which brings difficulties to long-distance drone monitoring and detection.

[0003] Super Resolution (SR) refers to the reconstruction of a corresponding high-resolution image from an observed low-resolution image through software or hardware methods. The reconstructed image contains the same scene, object and semantic information as the original image. Super-resolution technology has important application value in monitoring equipment, satellite image remote sensing, digital high-definition, microscopic imaging, video coding communication, video restoration and medical imaging. Through super-resolution, a clearer drone structure can be obtained from low-resolution drone image signals, which can improve the system's monitoring and countermeasure capabilities against rotary-wing drones.

[0004] Currently, there is no super-resolution method for rotary-wing drones. Patent CN113379597A introduces the prior information of key points of the face and the attention mechanism based on the SRFBN network, and designs a super-resolution reconstruction network for the face. This method feeds back the key point heat map to the super-resolution module and achieves super-resolution through loop iteration. Since key point detection and loop iteration require a lot of computational overhead in the inference stage, the operation efficiency of this method is low and it is difficult to implement terminal deployment applications. Patent CN114463174A proposes an image super-resolution method based on meta-learning and structure preservation. This method uses the network to process the original image and gradient map respectively during inference, and finally obtains a high-resolution image by fusing the image information and gradient information. Since this method uses different network structures for processing the original image and gradient map, there is also a lot of computational overhead. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a structure-guided rotorcraft UAV image super-resolution network construction method, which makes full use of the UAV gradient and key point information to reconstruct an accurate and clear UAV structure, while reducing the computational overhead during network inference and improving operating efficiency.

[0006] To achieve the above object, according to a first aspect of the present invention, a structure-guided rotorcraft UAV image super-resolution network construction method is provided, comprising:

[0007] A structure-guided rotorcraft UAV image super-resolution network is constructed, comprising: a preprocessing module for preprocessing a UAV low-resolution image, wherein the preprocessing includes image normalization; a gradient calculation module for obtaining a gradient map of the preprocessed UAV low-resolution image; a fusion module for performing channel-level fusion of the preprocessed low-resolution image and the gradient map to obtain a fused image; a feature extraction network for extracting features of the fused image to obtain a low-resolution feature map; a super-resolution module for upsampling the low-resolution feature map to obtain a high-resolution feature map; a key point prediction module, a high-resolution gradient map prediction module and a high-resolution image prediction module, respectively used to predict various key point heat maps, high-resolution gradient maps and high-resolution images of the UAV according to the high-resolution feature map; a discriminator for calculating the probability that the high-resolution image generated by the high-resolution image prediction module is a real image when training the network;

[0008] The low-resolution image of the drone is used as a sample, and various key point heat maps, high-resolution gradient maps and high-resolution images corresponding to the low-resolution image of the drone are used as labels to perform supervised training on the super-resolution network to obtain a trained super-resolution network.

[0009] According to a second aspect of the present invention, a structure-guided rotorcraft UAV image super-resolution network construction method is provided, comprising:

[0010] The low-resolution image of the drone to be processed is input into the super-resolution network constructed by the method described in the first aspect to obtain a high-resolution image of the low-resolution image of the drone to be processed.

[0011] According to a third aspect of the present invention, there is provided an electronic device, comprising: a computer-readable storage medium and a processor;

[0012] The computer-readable storage medium is used to store executable instructions;

[0013] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the method according to the first aspect or the method according to the second aspect.

[0014] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method as described in the first aspect or the method as described in the second aspect.

[0015] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0016] 1. The method provided by the present invention uses gradient and key points, two features that can characterize the structure of the drone, to guide the super-resolution of the rotorcraft drone, thereby improving the accuracy of super-resolution reconstruction; and constructs a multi-task learning network, which uses lightweight branches to complete the prediction of high-resolution gradient maps, key point predictions, and high-resolution drone predictions respectively. During reasoning, only the high-resolution drone prediction branch needs to be used to reconstruct a clear drone structure, saving a lot of computational overhead and being highly efficient. The multi-task learning network is a structure-guided rotorcraft drone super-resolution network, which performs channel machine fusion on low-resolution images and their low-resolution gradient maps and is processed by a feature extraction network to obtain a feature map, inputs the feature map into the key point prediction module of the network to output a low-resolution key point heat map, and then inputs the above feature map into the super-resolution processing module to further obtain a high-resolution drone gradient map and a high-resolution drone image, and performs loss calculation and supervision constraints on the high-resolution gradient map, the low-resolution key point heat map, and the real label results respectively, and the discriminator determines the probability that the output image is a real image to enhance the reconstruction ability of the super-resolution network, and the network is trained until the network converges. The network can output super-resolution images of rotorcraft UAVs with good visual effects, which improves the detection and monitoring effects of long-range rotorcraft UAV imaging.

[0017] 2. The method provided by the present invention, when constructing the training data set of the above-mentioned multi-task learning network, performs image enhancement and screening on the high-resolution UAV original image, and crops and resizes it according to the UAV position and annotation information, and then performs corresponding processing to obtain the UAV low-resolution sample image and the corresponding various key point heat map labels, high-resolution gradient map labels and high-resolution image labels, which can further improve the visibility of the target and the retention of texture details after the high-resolution image is downsampled, and ensure the basis for super-resolution reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the overall process of the structure-guided rotorcraft UAV super-resolution network construction method provided by an embodiment of the present invention;

[0019] Figure 2A schematic diagram of the training process of a super-resolution network for a rotary-wing UAV provided in an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the inference process of the super-resolution network of a rotary-wing UAV provided in an embodiment of the present invention;

[0021] Figure 4 A structural diagram of a super-resolution network of a rotary-wing UAV provided in an embodiment of the present invention;

[0022] Figure 5 An example diagram of a feature extraction network in a super-resolution network of a rotary-wing UAV provided in an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of a residual network used in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0025] In addition to the high computational overhead, existing super-resolution methods only utilize key point information and gradient information respectively. Although only using key point information can focus on the local features of the target image to a certain extent, it is easy to ignore the overall structural information of the image, and the generated high-resolution image may be distorted or the key area may be blurred. On the other hand, using only gradient information may make the model framework focus more on the reconstruction of edges and details, ignoring the accurate restoration of key parts, resulting in distortion of key part features or inconsistency with the real image. Such processing methods usually cannot take into account both the global structure and detail information of the image at the same time, which limits the quality of super-resolution reconstruction.

[0026] Based on this, an embodiment of the present invention provides a structure-guided rotorcraft UAV image super-resolution network construction method, comprising:

[0027] A structure-guided rotorcraft UAV image super-resolution network is constructed, comprising: a preprocessing module for preprocessing a UAV low-resolution image, wherein the preprocessing includes image normalization; a gradient calculation module for obtaining a gradient map of the preprocessed UAV low-resolution image; a fusion module for performing channel-level fusion on the preprocessed low-resolution image and the gradient map to obtain a fused image; a feature extraction network for extracting features of the fused image to obtain a low-resolution feature map; a super-resolution module for upsampling the low-resolution feature map to obtain a high-resolution feature map; a key point prediction module, a high-resolution gradient map prediction module and a high-resolution image prediction module, respectively used to predict various key point heat maps, high-resolution gradient maps and high-resolution images of the UAV according to the high-resolution feature map; and a discriminator for calculating the probability that the high-resolution image generated by the high-resolution image prediction module is a real image when training the network.

[0028] The low-resolution image of the drone is used as a sample, and various key point heat maps, high-resolution gradient maps and high-resolution images corresponding to the low-resolution image of the drone are used as labels to perform supervised training on the super-resolution network to obtain a trained super-resolution network.

[0029] Specifically, after constructing the structure-guided rotorcraft UAV super-resolution network, it is necessary to construct a data set to train the structure-guided rotorcraft UAV super-resolution network. The sample images in the data set are low-resolution images of the UAV, and the label images are various key point heat maps, high-resolution gradient maps and high-resolution images corresponding to the sample images.

[0030] It can be understood that the corresponding low-resolution image of the drone can be obtained by downsampling the high-resolution rotorcraft drone original image, and the corresponding various key point heat maps and high-resolution gradients can be obtained through heat map calculation and gradient calculation, and the high-resolution rotorcraft drone original image can be used as a high-resolution image.

[0031] In order to further improve the visibility of the target and the retention of texture details after the high-resolution image is downsampled, and to ensure the basis for super-resolution reconstruction, as a further preferred embodiment of the present invention, the acquisition method of the drone low-resolution image and its corresponding various key point heat maps, high-resolution gradient maps and high-resolution images is as follows:

[0032] After performing image enhancement on the captured high-resolution drone original image, an image whose drone target detection frame pixels exceed a pixel threshold is selected as a first target image, the position coordinates and key point coordinates of the drone in the first target image are annotated, and the first target image is cropped according to the annotated information so that only the drone area is left to obtain a second target image, the second target image is resized so that its size meets the requirements, and then down-sampled to obtain a low-resolution image of the drone;

[0033] Generate a heat map of each key point according to the coordinates of each key point in the first target image, superimpose the heat maps of key points of the same category, and then perform corresponding cropping and transformation on each type of superimposed heat map according to the size of the second target image to obtain heat maps of each type of key point corresponding to the low-resolution image of the drone;

[0034] The second target image is a high-resolution image corresponding to the low-resolution image of the drone; and the gradient map of the high-resolution image is a high-resolution gradient map corresponding to the low-resolution image of the drone.

[0035] Specifically, first, the original image of the rotary-wing UAV is collected. The high-resolution original image of the UAV is an RGB image or a grayscale image. For example, if the high-resolution original image of the UAV is an RGB image, a color three-channel rotary-wing UAV image under different light conditions is captured. The embodiment of the present invention is described by taking the high-resolution original image of the UAV as an RGB image as an example.

[0036] After collecting the images of the rotary-wing drone, the images need to be enhanced. Image enhancement operations include rotation, translation, cropping, flipping, adding noise, etc. These operations are performed randomly to generate more diverse variants, making the data closer to various data in real scenes and improving the coverage of the data.

[0037] Then, the pictures that meet the preset conditions in the enhanced image of the rotorcraft UAV are selected to calibrate the position and key points of the UAV. The preset conditions are: the pixels of the UAV target detection frame exceed the pixel threshold, and the pixel threshold can be set according to the actual situation, for example: 200×200.

[0038] After the specific position of the rotary-wing UAV in the image is obtained and marked, key points of various parts of the rotary-wing UAV are marked.

[0039] It is understandable that the key points of the rotary-wing UAV are the center points of the top of each rotor of the UAV and the center point of the battery part of the rotary-wing UAV fuselage.

[0040] Taking a quad-rotor drone as an example, the key points of each part of the rotor drone include: the top center point of the four rotors and the center point of the battery part of the rotor drone body.

[0041] The position and key point annotation information of the above-mentioned quadcopter can be expressed by a file in the following format: [({'bbox':array([x_1,y_1,x_2,y_2],dtype=int64)}),(img_name,{'landmark':array([[x1,y1],[x2,y2],[x3,y3],[x4,y4],[x5,y5]])})]. In which, x_1,y_1 represent the pixel coordinates of the upper left corner of the position of the rotorcraft, and x_2,y_2 represent the pixel coordinates of the lower right corner of the position of the rotorcraft. x1,x2,x3,x4 are the horizontal coordinates of the top of the rotor of the quadcopter, and y1,y2,y3,y4 are the vertical coordinates of the top of the rotor of the quadcopter; x5 is the horizontal coordinate of the battery part of the quadcopter fuselage, and y5 is the vertical coordinate of the battery part of the quadcopter fuselage.

[0042] According to the position of the rotary-wing UAV in the image and the corresponding key point annotation information, the pixel coordinates of each key point in the corresponding rotary-wing UAV image are obtained, and a heat map of each key point is constructed. The construction method is as follows:

[0043]

[0044] In this embodiment, the Gaussian peak parameters are set as follows: the standard deviation σ in the xy direction is 25, the maximum value of the Gaussian peak is 255, (x, y) is the coordinate of the key point, and (i, j) is the coordinate of each pixel of the key point heat map.

[0045] Taking a quadcopter as an example, since a quadcopter has four rotors corresponding to four key points, and one key point corresponds to a heat map, multiple Gaussian peaks are superimposed on a heat map in the heat map of such key points, that is, the heat maps of the center points of all rotor tips are superimposed into one heat map.

[0046] The following processing is performed on the pictures that meet the preset conditions in the original image of the rotary-wing UAV to obtain a three-channel RGB image of the low-resolution rotary-wing UAV that meets the input requirements of the super-resolution network, specifically including:

[0047] The high-resolution drone image (i.e., the image that meets the preset conditions in the original image of the rotary-wing drone) is cropped, and the background in the image is cropped based on the drone position annotation information, leaving only the drone area; then, the cropped high-resolution drone image is resized to make its size uniform (the specific uniform size can be set according to actual requirements, and the embodiment of the present invention is described by taking the uniform size of 256×256 as an example), and then downsampled to a three-channel RGB color image of a specific width and length to obtain a low-resolution image; the key point information and key point heat map annotated by the obtained low-resolution rotary-wing drone image are transformed accordingly to obtain a low-resolution drone image and related key point information, and the transformation formula is:

[0048]

[0049] Among them, x i is the horizontal coordinate of the point in the cropped area at the original size, x i ′ is the horizontal axis coordinate after cropping and downsampling, k x is the horizontal axis downsampling factor (k x >1), The starting horizontal coordinate of the image cropping, scale x is the scale factor, which is equal to the image width after resize (i.e., resize transformation) divided by the image width after crop (i.e., cropping) multiplied by the inverse of the sampling rate. i ′ The calculation of y is similar. i is the vertical coordinate of the point in the cropped area at the original size, y i ′ is the vertical axis coordinate after cropping and downsampling, k y is the vertical axis downsampling factor (k y >1), The starting vertical coordinate for image cropping.

[0050] As an example, in this embodiment, the resize value is set to 256×256, and the downsampling ratio is set to 4 times.

[0051] like Figure 4 As shown, the structure-guided rotorcraft UAV super-resolution network includes a gradient calculation module, a fusion module, a feature extraction network, a super-resolution module, a key point prediction module, a high-resolution gradient map prediction module, a high-resolution image prediction module and a discriminator.

[0052] The following introduces the various modules of the network in conjunction with the training process of the structure-guided rotorcraft super-resolution network. The training process of the structure-guided rotorcraft super-resolution network includes the following steps:

[0053] (1) Before the low-resolution image data of the drone is input into the network, it needs to be preprocessed to standardize the data and adapt it to the input requirements of the network model. The preprocessing operation includes image normalization to adjust the image pixels to a fixed range to facilitate the subsequent calculation of the network; further, the preprocessing operation also includes color space conversion and / or denoising; among them, when the network model processes visible light data and infrared data for training and testing at the same time, color space conversion can be performed to convert the image color space into RGB format so that the infrared single-channel image can be processed as a color image.

[0054] (2) The gradient calculation module processes the low-resolution image to obtain a gradient map.

[0055] The Sobel operator is usually used to extract gradients in the x and y directions. When extracting gradients from a multi-channel image (such as an RGB image), the RGB image must first be converted to a single-channel grayscale image. This conversion combines color information into a brightness value using the following conversion formula:

[0056] I=0.2989·R+0.5870·G+0.1140·B (4)

[0057] The following calculations are then required:

[0058] I X (x)=I(x+1,y)-I(x-1,y), (5)

[0059]

[0060] Among them, the coordinate (x, y) is the position of a pixel in the low-resolution image, I(x, y) is the pixel value, and M(·) represents the operation of extracting the gradient map, whose element is the gradient length of the pixel at the coordinate x = (x, y). This operation can be easily implemented by a convolution layer with a fixed convolution kernel.

[0061] (3) The fusion module performs channel-level fusion on the low-resolution UAV image and gradient map and outputs the fused data.

[0062] When performing image channel-level fusion, the input low-resolution drone image and gradient map Connect channels to get fused data Where M is the image width, N is the image length, and c is the number of input low-resolution drone image channels. Specifically, the high-resolution drone original image is an RGB image or a grayscale image, and c is set to 3 and 1 respectively.

[0063] (4) After the channel-level fusion data of the low-resolution UAV image and the low-resolution gradient map are input into the feature extraction network, a low-resolution UAV feature map is output.

[0064] The feature extraction network uses an improved HRNet network. On the basis of the HRNet network, the input dimension of the first layer is set to c+1, the feature dimension of the network is set to 256, the low-resolution module in the last layer of feature extraction module is removed, and only the highest-resolution module is retained to output the drone feature map. Where M is the image width, N is the image length, and D is the feature dimension, which can be set according to actual conditions. For example, it can be set to 256. The network structure of the feature extraction module is as follows: Figure 5 shown.

[0065] (5) The low-resolution UAV feature map is input into the key point prediction module to predict the key points of the UAV. The output form of the key points is a heat map.

[0066] The key point prediction module consists of a residual module and a 1×1 convolution, where the input and output dimensions of the residual module and the input dimension of the 1×1 convolution are both 256, and the output dimension of the 1×1 convolution is the number of key point categories. Usually, the key point categories are the rotor center point and the drone battery center point, that is, the number of key point categories is 2.

[0067] Heatmap is a commonly used key point representation method, which is used to represent the location of key points and their confidence. Heatmap is usually generated by mapping the location of key points onto a two-dimensional grid and using Gaussian distribution around the key point location. The formula is as follows:

[0068]

[0069] In network training, the coordinates of the key points returned after regression tend to be overfitted on the training set. Therefore, using this method, a heat map with the number of channels equal to the number of key point categories can be returned to train a robust network.

[0070] (6) The low-resolution UAV feature map is input into the super-resolution module, which outputs the UAV high-resolution feature map.

[0071] The super-resolution module contains two upsampling modules, each of which consists of a residual module and a PixelShuffle layer. The input and output dimensions of the residual module and a PixelShuffle layer are both 256.

[0072] Among them, the specific settings of the residual module are: a convolution kernel of 1×1, the number of input channels is 256, and the number of output channels is 128; a batch normalization layer with an input channel number of 128; a ReLU activation layer. A convolution kernel size of 3×3, the number of input channels is 128, the number of output channels is 128, the padding is 1, and the bias is set to True; a batch normalization layer with an input channel number of 128, and a ReLU activation layer. A convolution kernel size of 1*1, the number of input channels is 128, the number of output channels is 256, and the bias is set to True. In addition, the final output tensor needs to be superimposed with the initial input tensor. See for details. Figure 6 .

[0073] (7) The high-resolution feature map of the UAV is input into the high-resolution gradient map prediction module, which outputs the high-resolution UAV gradient map.

[0074] The high-resolution gradient map prediction module includes a residual module and a 1×1 convolution. The input and output dimensions of the residual module and the input dimension of the 1×1 convolution are both 256, and the output dimension of the 1×1 convolution is 1.

[0075] (8) The high-resolution feature map of the UAV is input into the high-resolution image prediction module, which outputs the high-resolution UAV image.

[0076] The high-resolution image prediction module includes the above residual module and a 1×1 convolution. The input and output dimensions of the residual module and the input dimension of the 1×1 convolution are both 256, and the output dimension of the 1×1 convolution is 3. It can be understood that when the original high-resolution drone image is a grayscale image, the output dimension of the 1×1 convolution is 1.

[0077] (9) Compare the gradient map and key point heat map of the real high-resolution drone image with the high-resolution drone gradient map obtained in step (7) and the key point heat map output in step (5), and calculate the loss. The gradient map reflects the structure and contour edges of the drone, while the key points represent the fine-grained representation of the overall structure of the drone. These two serve as drone structural constraints and contribute to the super-resolution reconstruction of drone images.

[0078] The loss calculation includes: obtaining the heat map label G of each key point g , get the high-resolution drone label H g , get the high-resolution gradient map label G g ; The loss is calculated according to the following formula:

[0079] loss = λ g ‖G g -G p ‖+λ h ‖Hg -H p ‖+λ heat ‖Heat g -Heat p ‖+λ adv l d , (10)

[0080] Among them, λ g , h , heat , adv are the weight parameters of the heat map, high-resolution drone, high-resolution gradient map and discriminator loss, respectively, ‖·‖ is the F-1 norm; l d is the discriminator loss, calculated by cross entropy loss, and the calculation formula is as follows:

[0081] L D =-Log(D(I HR ))-Log(1-D(G(I LR ))), (11)

[0082] Among them, D(I HR ) represents the predicted probability of the discriminator for the real image, D(G(I LR )) represents the predicted probability of the discriminator for the generated image.

[0083] As an example, the initial value of the learning rate of this network is set to 0.01, and it is multiplied by 0.1 every 20 epochs; the optimizer uses ADAM, and the parameter is set to β 1 =0.9,β 2 =0.999,ε=1×10 -8 ; The hyperparameter for loss calculation is set to λ g =λ h =λ heat =1,λ adv =0.001, and the number of training rounds is 100.

[0084] (10) The generated high-resolution UAV image and the real high-resolution UAV image are input into the discriminator, and the probability of the output image being a real image is output. This module can promote the rotorcraft UAV super-resolution network to generate more realistic high-resolution images.

[0085] It can be understood that after the network is trained, the network in inference mode inputs low-resolution rotorcraft drone images, and the output is super-resolution drone images.

[0086] The embodiment of the present invention provides a structure-guided rotorcraft UAV image super-resolution network construction method, comprising:

[0087] The low-resolution image of the drone to be processed is input into the super-resolution network constructed by the method described in any of the above embodiments to obtain a high-resolution image of the low-resolution image of the drone to be processed.

[0088] An embodiment of the present invention provides an electronic device, characterized by comprising: a computer-readable storage medium and a processor;

[0089] The computer-readable storage medium is used to store executable instructions;

[0090] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method described in any of the above embodiments.

[0091] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0092] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A structure-guided super-resolution network construction method for rotary-wing UAV images, It is characterized in that include: A structure-guided rotorcraft UAV image super-resolution network is constructed, comprising: a preprocessing module for preprocessing a UAV low-resolution image, wherein the preprocessing includes image normalization; a gradient calculation module for obtaining a gradient map of the preprocessed UAV low-resolution image; a fusion module for performing channel-level fusion of the preprocessed low-resolution image and the gradient map to obtain a fused image; a feature extraction network for extracting features of the fused image to obtain a low-resolution feature map; a super-resolution module for upsampling the low-resolution feature map to obtain a high-resolution feature map; a key point prediction module, a high-resolution gradient map prediction module and a high-resolution image prediction module, respectively used to predict various key point heat maps, high-resolution gradient maps and high-resolution images of the UAV according to the high-resolution feature map; a discriminator for calculating the probability that the high-resolution image generated by the high-resolution image prediction module is a real image when training the network; The low-resolution image of the drone is used as a sample, and various key point heat maps, high-resolution gradient maps and high-resolution images corresponding to the low-resolution image of the drone are used as labels to perform supervised training on the super-resolution network to obtain a trained super-resolution network.

2. The method according to claim 1, It is characterized in that The preprocessing also includes color space conversion and / or denoising.

3. The method according to claim 1 or 2, It is characterized in that The method for obtaining the low-resolution image of the drone and the corresponding various key point heat maps, high-resolution gradient maps and high-resolution images is as follows: After image enhancement is performed on the high-resolution drone original image, images whose drone target detection frame pixels exceed the pixel threshold are selected as the first target image, the position coordinates and key point coordinates of the drone in the first target image are marked, and the first target image is cropped according to the position marking information so that only the drone area is left to obtain the second target image, the second target image is resized to make its size meet the requirements, and then down-sampled to obtain the low-resolution image of the drone; Generate a heat map of each key point according to the coordinates of each key point in the first target image, superimpose the heat maps of key points of the same category, and then perform corresponding cropping and transformation on each type of superimposed heat map according to the size of the second target image to obtain heat maps of each type of key point corresponding to the low-resolution image of the drone; The second target image is a high-resolution image corresponding to the low-resolution image of the drone; and the gradient map of the high-resolution image is a high-resolution gradient map corresponding to the low-resolution image of the drone.

4. The method according to claim 3, It is characterized in that The high-resolution drone original image includes an RGB image or a grayscale image.

5. The method according to claim 1, It is characterized in that The loss function of the super-resolution network is: loss=λ g ‖G g -G p ‖+λ h ‖H g -H p ‖+λ heat ‖Heat g -Heat p ‖+λ adv l d Among them, λ g , h , heat , adv are the weight parameters of the heat map, high-resolution drone, high-resolution gradient map and discriminator loss, respectively, ‖·‖ is the F-1 norm; l d is the discriminator loss.

6. The method according to claim 1, It is characterized in that The feature extraction module is obtained by removing the low-resolution module in the last layer feature extraction module of the HRNet network.

7. The method according to claim 1, It is characterized in that The key point prediction module includes a residual module connected in sequence and a convolution layer whose output dimension is the number of key point categories; The high-resolution gradient map prediction module includes a residual module and a convolutional layer with an output dimension of 1 connected in sequence; The high-resolution image prediction module includes a residual module and a convolutional layer with an output dimension of K connected in sequence; wherein K is the number of channels of the high-resolution drone original image.

8. A structure-guided super-resolution network construction method for rotary-wing UAV images. It is characterized in that include: The low-resolution image of the drone to be processed is input into the super-resolution network constructed by the method described in any one of claims 1 to 7 to obtain a high-resolution image of the low-resolution image of the drone to be processed.

9. An electronic device, It is characterized in that include: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium, and execute the method according to any one of claims 1 to 7 or the method according to claim 8.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 7 or the method according to claim 8.

Citation Information

Patent Citations

  • Multi-task super-resolution image reconstruction method based on KSVD dictionary learning

    CN101950365A

  • Any-scale image super-resolution method based on meta-learning and structure preserving

    CN114463174A

  • Face image super-resolution method based on heat map prior guidance

    CN116309048A

  • Generative Adversarial Network for Dental Image Super-Resolution, Image Sharpening, and Denoising

    US20210118099A1

  • Ultra-high resolution CT reconstruction using gradient guidance

    WO2023051344A1

Cited By

  • Defect detection method and system for gantry machine tool workbench casting part

    CN120927812A