Satellite image super-resolution reconstruction method based on unmanned aerial vehicle image assistance

Abstract

The embodiment of the invention discloses a satellite image super-resolution reconstruction method based on unmanned aerial vehicle image assistance. The method comprises the following steps: acquiring an aerial image acquired by an unmanned aerial vehicle, performing aerial triangulation based on an image control point, and performing ortho-rectification on the aerial image; selecting first images of different terrain areas, performing down-sampling on the first images to obtain second images, and constructing first training data to train a generative adversarial network; and obtaining a to-be-reconstructed target image, reading the target image into at least two sub-blocks in a blocking manner, generating a reconstruction result of each sub-block based on the trained generative adversarial network, and splicing the reconstruction results based on an edge gradient fusion method to obtain a reconstructed image. According to the embodiment of the invention, an image with higher resolution can be obtained without lifting or replacing hardware, the cost is saved, and the technology is easy to realize.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to image reconstruction technology, and more particularly to a method for super-resolution reconstruction of satellite images based on the assistance of drone images. Background Art

[0002] Remote sensing satellite images play an important role in land and resources monitoring, urban planning, agricultural production and other fields. However, due to factors such as imaging equipment, atmospheric conditions and orbital altitude, the spatial resolution of satellite images is low and often cannot meet the needs of refined applications. Although higher resolution images can be obtained by improving the performance of sensor hardware, this solution is costly and technically difficult to implement. Therefore, there is a need for a method that can reconstruct low-resolution satellite images at a super-resolution. Summary of the invention

[0003] In order to solve the above problems, one or more embodiments of this specification describe a satellite image super-resolution reconstruction method based on drone image assistance.

[0004] According to a first aspect, a method for super-resolution reconstruction of satellite images based on drone image assistance is provided, the method comprising: Acquire aerial images collected by drones, perform aerial triangulation based on image control points set in the aerial images, and orthorectify the aerial images according to a digital elevation model generated by the measurement results to obtain a digital orthophoto map; Selecting first images of different terrain areas in the digital orthophoto map, downsampling the first images to obtain second images, constructing first training data pairs based on the first images and the second images, and training a generative adversarial network based on the first training data pairs, wherein each of the first training data pairs includes a high-resolution image and a low-resolution image with the same area size and area coordinates; Obtain a target image to be reconstructed, divide the target image into at least two sub-blocks, generate reconstruction results of each sub-block based on a trained generative adversarial network, and splice the reconstruction results based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image.

[0005] Preferably, the acquiring of aerial images collected by a drone and performing aerial triangulation based on image control points set in the aerial images include: Acquire aerial images collected by a drone based on oblique photography, and display the aerial images; A setting instruction for the aerial image is received, image control points in the aerial image are determined, and aerial triangulation is performed based on each of the image control points to obtain measurement data.

[0006] Preferably, the digital elevation model generated according to the measurement results is used to orthorectify the aerial image to obtain a digital orthophoto map, including: Projecting pixel coordinates in the aerial image into three-dimensional coordinates in a three-dimensional space according to the measurement results, determining three-dimensional point cloud data in the three-dimensional space, and constructing a digital elevation model according to the three-dimensional point cloud data; After optimizing each of the three-dimensional coordinates based on the ground elevation value of the digital elevation model, resampling the optimized three-dimensional coordinates to obtain new pixel coordinates, and generating a preliminary orthophoto map corresponding to each of the aerial images based on the new pixel coordinates; The preliminary orthophoto images are stitched together to obtain a digital orthophoto image.

[0007] Preferably, constructing a first training data pair based on the first image and the second image includes: After adding Gaussian noise to the second image, Gaussian blur processing is performed on the second image to obtain a third image; Cutting the first image and the third image into at least two first sub-images and a second sub-image respectively, wherein the first sub-image and the second sub-image have the same image size; The first sub-image and the second sub-image corresponding to the same image coordinates are constructed as a first training data pair, and the first training data pairs are integrated to obtain a training data set.

[0008] Preferably, the integrating the first training data pairs to obtain a training data set includes: Acquire a regional satellite image of each of the terrain areas, and adjust the regional satellite image based on geometric correction so that the geographical location of the regional satellite image is consistent with the first image; constructing a second training data pair based on the first image and a regional satellite image; The first training data pairs and the second training data pairs are integrated to obtain a training data set.

[0009] Preferably, the training of a generative adversarial network based on the first training data comprises: Obtain an initial generative adversarial network, wherein the generator of the initial generative adversarial network is a deep network with at least two dense residual learning blocks, and the discriminator of the initial generative adversarial network is a U-Net discriminator with skip connections; The initial generative adversarial network is trained based on the first training data pair to obtain a preliminary training network, and the preliminary training network is optimized based on the second training data pair to obtain a trained generative adversarial network.

[0010] Preferably, the step of obtaining a target image to be reconstructed, reading the target image into at least two sub-blocks, generating reconstruction results of each sub-block based on a trained generative adversarial network, and splicing each reconstruction result based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image includes: Acquire a target image to be reconstructed, and read the target image into at least two sub-blocks, where there is an overlapping area of ​​a preset number of pixels between adjacent sub-blocks; Generate a reconstruction result of each sub-block based on a trained generative adversarial network; For overlapping pixels corresponding to overlapping areas in the reconstruction results, the weighted pixel sum of edge distances of overlapping pixels at the same position in each reconstruction result is taken as the final pixel value of the position to obtain a reconstructed image corresponding to the target image.

[0011] According to a second aspect, a satellite image super-resolution reconstruction device based on drone image assistance is provided, the device comprising: An acquisition module is used to acquire aerial images collected by a drone, perform aerial triangulation based on image control points set in the aerial images, and orthorectify the aerial images according to a digital elevation model generated by the measurement results to obtain a digital orthophoto map; A training module, configured to select first images of different terrain areas in the digital orthophoto map, obtain second images after downsampling the first images, construct first training data pairs based on the first images and the second images, and train a generative adversarial network based on the first training data pairs, wherein each of the first training data pairs includes a high-resolution image and a low-resolution image with the same area size and area coordinates; The reconstruction module is used to obtain the target image to be reconstructed, read the target image into at least two sub-blocks, generate reconstruction results of each sub-block based on the trained generative adversarial network, and splice the reconstruction results based on the edge gradient fusion method to obtain the reconstructed image corresponding to the target image.

[0012] According to a third aspect, there is provided an electronic device, comprising a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the steps of the method provided in the first aspect or any possible implementation manner of the first aspect.

[0013] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer-readable storage medium stores instructions, which, when the instructions are executed on a computer or a processor, cause the computer or the processor to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0014] The method provided in the embodiments of this specification can obtain a digital orthophoto map after orthophoto processing based on the aerial images collected by the drone, and obtain a first image used to represent the high-resolution image and a second image used to represent the low-resolution image based on the digital orthophoto map, construct a first training data pair to train the generative adversarial network, and finally achieve super-resolution reconstruction of the target image based on the generative adversarial network. Through the above method, the low-resolution drone image after downsampling processing can be used as a low-resolution satellite image, and the training data pair can be combined with the high-resolution drone image to train the generative adversarial network, so that the trained generative adversarial network can perform super-resolution reconstruction of the satellite image, and higher-resolution images can be obtained without upgrading or replacing hardware, which saves costs and is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flowchart of a method for super-resolution reconstruction of satellite images based on drone image assistance in one embodiment of the present specification.

[0017] Figure 2 It is a structural schematic diagram of a satellite image super-resolution reconstruction device based on drone image assistance in one embodiment of the present specification.

[0018] Figure 3 It is a structural diagram of an electronic device in one embodiment of this specification. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0020] In the following introduction, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following introduction provides multiple embodiments of the present application, and different embodiments can be replaced or combined, so the present application can also be considered to include all possible combinations of the same and / or different embodiments recorded. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, although the embodiment may not be clearly recorded in the following text.

[0021] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes may be made to the functions and arrangements of the elements described without departing from the scope of the present application. Various processes or components may be appropriately omitted, substituted or added to each example. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted or combined. In addition, features described in some examples may be combined in other examples.

[0022] See also Figure 1 , Figure 1 : is a flowchart of a method for super-resolution reconstruction of satellite images based on drone image assistance provided in an embodiment of the present application. In an embodiment of the present application, the method includes: S101, obtaining an aerial image collected by a drone, performing aerial triangulation based on image control points set in the aerial image, and orthorectifying the aerial image according to a digital elevation model generated by the measurement result to obtain a digital orthophoto map.

[0023] The execution entity of this application can be a cloud server.

[0024] In the embodiment of this specification, in order to construct a training set, the drone is first controlled to take aerial photos to obtain aerial images collected by the drone. Compared with satellites, the shooting distance of drones from the ground is lower, so the aerial images captured are more high-definition, and these aerial images will be used as high-resolution images in subsequent training sets. In order to allow aerial images to be used to characterize the situation of satellite images, before using aerial images as high-resolution images of the training set, the aerial images need to be pre-processed and adjusted to orthophotos that match the satellite images. Specifically, after the drone collects the aerial images, the staff can lay out the image control points of the aerial images, that is, select some easily identifiable and difficult to change objects (such as road intersections, building corners, obvious terrain features, etc.) in the image as image control points, and mark them in the aerial images. The aerial images obtained by the cloud server can be images that have been marked, so the image control points set by the staff can be directly determined from the images. Through the image control points, aerial triangulation can be performed. Aerial triangulation (AT) is a method of determining the geometric relationship between images through photogrammetry technology. It will find the same image control points in different images through feature matching algorithms, and use the least squares method or other optimization methods to calculate the internal and external orientation elements of each image, that is, the measurement results, and then construct a three-dimensional space model of the corresponding position of the image to obtain the specific three-dimensional data of the corresponding position. Then, a digital elevation model can be generated, and then the three-dimensional space model can be adjusted through the reverse optimization of the digital elevation model. Finally, each pixel in the aerial image is corresponded to an exact position on the ground through the three-dimensional space model, and orthorectification is achieved to obtain the digital orthophoto map corresponding to the aerial image. Among them, after aerial triangulation, the aerial image can also be radiometrically corrected at the same time to eliminate brightness changes caused by factors such as atmospheric scattering.

[0025] In one possible implementation, the step of acquiring the aerial image collected by the drone and performing aerial triangulation based on the image control points set in the aerial image includes: Acquire aerial images collected by a drone based on oblique photography, and display the aerial images; A setting instruction for the aerial image is received, image control points in the aerial image are determined, and aerial triangulation is performed based on each of the image control points to obtain measurement data.

[0026] In the embodiment of this specification, the drone will first collect aerial images by oblique photography. Oblique photography is to carry multiple sensors on the same drone and collect images from five different angles, such as a vertical, four oblique, etc., to obtain richer high-resolution textures of the top and side views of the building, and improve the accuracy of subsequent processing results. After the aerial images are collected, they will be displayed to the terminal used by the staff, and the staff can set the image control points after quality inspection of the images. The cloud server will receive the setting instructions generated by the staff operation to determine the image control points set by the staff, and then perform aerial triangulation based on the image control points to obtain measurement data. Among them, after obtaining the measurement data, the measurement data can also be sent to the staff again, and the staff will perform quality inspection and screening on the measurement data.

[0027] In one possible implementation, the digital elevation model generated according to the measurement result performs orthorectification on the aerial image to obtain a digital orthophoto map, including: Projecting pixel coordinates in the aerial image into three-dimensional coordinates in a three-dimensional space according to the measurement results, determining three-dimensional point cloud data in the three-dimensional space, and constructing a digital elevation model according to the three-dimensional point cloud data; After optimizing each of the three-dimensional coordinates based on the ground elevation value of the digital elevation model, resampling the optimized three-dimensional coordinates to obtain new pixel coordinates, and generating a preliminary orthophoto map corresponding to each of the aerial images based on the new pixel coordinates; The preliminary orthophoto images are stitched together to obtain a digital orthophoto image.

[0028] In the embodiment of the present specification, the measurement result includes the internal orientation elements (such as focal length, principal point coordinates) and external orientation elements (such as coordinates and attitude angles of the image center) of each image calculated by the relationship between the control points and the overlapping images. By solving the measurement results, the pixel coordinates in the aerial image can be projected into three-dimensional coordinates in three-dimensional space to construct a three-dimensional space model so that each image has an accurate positioning in three-dimensional space. At the same time, after the three-dimensional space model is constructed by aerial triangulation, a large amount of three-dimensional point cloud data will be generated. These point cloud data contain the height information of the surface, so a digital elevation model (Digital Elevation Model, DEM) can be constructed based on these height information. The height coordinate parameters in the three-dimensional coordinates can be optimized and adjusted by the ground elevation value in the digital elevation model to achieve the refinement and adjustment of the three-dimensional model, so that it can more accurately reflect the actual terrain. By resampling the optimized three-dimensional coordinates, the new pixel coordinates corresponding to the aerial image can be obtained, and the pixel values ​​of the new pixel coordinates are rewritten into the image file to obtain a preliminary orthophoto map. Aerial images taken by drones are smaller than satellite images, so the preliminary orthophotos taken need to be stitched together. If necessary, quality inspection and local correction can be performed manually to eventually form a digital orthophoto.

[0029] S102: selecting a first image of a different terrain area in the digital orthophoto map, downsampling the first image to obtain a second image, constructing a first training data pair based on the first image and the second image, and training a generative adversarial network based on the first training data pair.

[0030] Each of the first training data pairs includes a high-resolution image and a low-resolution image with the same region size and region coordinates.

[0031] In an embodiment of the present specification, in order to construct a training sample set, a first image of a certain size of different terrain areas (such as urban areas, rural areas, mountains, water towns, etc.) is selected from the digital orthophoto map as a high-resolution image of the drone image. Then, the first image is downsampled to obtain a second image as a low-resolution image of the drone image. The downsampling can specifically adopt a bicubic downsampling method, and the downsampling ratio can be selected as 4 times downsampling. The low-resolution second image is similar to the satellite remote sensing image collected by the satellite, so the images of the same position and size in the first image and the second image can be matched together to construct a first training data pair, and the generative adversarial network is trained based on the first training data pair, so that the trained generative adversarial network can output its corresponding high-resolution image when receiving the low-resolution satellite remote sensing image. Among them, when the first image and the second image are large, the first image and the second image can also be cut first, and then the first training data pair is constructed based on the cut images. In addition, since the second image has been downsampled, in order to make the area sizes corresponding to the high-resolution image obtained from the first image and the low-resolution image obtained from the second image in the first training data pair the same, the sizes of the high-resolution image and the low-resolution image in each first training data pair need to be adjusted according to the downsampling ratio. For example, assuming the downsampling is 4 times, the low-resolution image can be a 256*256 image, and the high-resolution image can be a 64*64 image.

[0032] In one possible implementation, constructing a first training data pair based on the first image and the second image includes: After adding Gaussian noise to the second image, Gaussian blur processing is performed on the second image to obtain a third image; Cutting the first image and the third image into at least two first sub-images and a second sub-image respectively, wherein the first sub-image and the second sub-image have the same image size; The first sub-image and the second sub-image corresponding to the same image coordinates are constructed as a first training data pair, and the first training data pairs are integrated to obtain a training data set.

[0033] In the embodiment of this specification, in order to make the quality of the low-resolution drone image closer to that of the satellite image, the second image is also degraded. Specifically, Gaussian noise is first added to the second image. For example, the calculation formula is as follows: in, is the original image, is Gaussian noise.

[0034] Next, the image with added noise is Gaussian blurred using a 3*3 convolution kernel: Among them, x and y are the offsets of the pixel position, is the standard deviation of the Gaussian function.

[0035] Next, the first image and the third image are cut to obtain a first sub-image corresponding to the first image and a second sub-image corresponding to the third image, and a first training data pair is constructed based on the first sub-image and the second sub-image. The first training data pair can be understood as a high-resolution-low-resolution data pair of drone images.

[0036] In one possible implementation, the integrating the first training data pairs to obtain a training data set includes: Acquire a regional satellite image of each of the terrain areas, and adjust the regional satellite image based on geometric correction so that the geographical location of the regional satellite image is consistent with the first image; constructing a second training data pair based on the first image and a regional satellite image; The first training data pairs and the second training data pairs are integrated to obtain a training data set.

[0037] In the embodiments of this specification, in order to further improve the accuracy of the trained model, in addition to the first training data pair constructed entirely from drone images, actual regional satellite images of various terrain areas are obtained, and the positions of the regional satellite images are adjusted through geometric correction, so that the regional satellite images can be basically consistent with the geographical location of the first image. Then, the first image and the regional satellite image are cut in the same way as the first training data pair, and a second training data pair representing drone high resolution and satellite low resolution is constructed. Among them, since the time phases of drone images and satellite images are not the same, it is necessary to manually screen the regional satellite images to eliminate image pairs that do not match or have large differences, such as the presence of new buildings, building demolition, dense vehicles, river vessels, etc. The first training data pair can be used to perform preliminary training on the model, and the second training data pair can be used to optimize and fine-tune the model.

[0038] In one possible implementation, the training of a generative adversarial network based on the first training data includes: Obtain an initial generative adversarial network, wherein the generator of the initial generative adversarial network is a deep network with at least two dense residual learning blocks, and the discriminator of the initial generative adversarial network is a U-Net discriminator with skip connections; The initial generative adversarial network is trained based on the first training data pair to obtain a preliminary training network, and the preliminary training network is optimized based on the second training data pair to obtain a trained generative adversarial network.

[0039] In the embodiments of this specification, the initial generative adversarial network model used in this application can be Real-ESRGAN, whose generator is a deep network with several dense residual learning blocks (RRDB), and all convolutional layers use a step size of 1 to maintain the resolution of the image in the generator. The concept of residual can prevent instability when training the deep network. The discriminator can use a U-Net discriminator with a jump connection. U-Net outputs the authenticity value of each pixel and can provide detailed feedback for each pixel to the generator. At the same time, in order to avoid the training instability caused by the U-Net network structure, frequency domain regularization can be used to stabilize the training dynamics, which also helps to alleviate the over-sharpening caused by the GAN training process. During the training process, the network is first trained by the first training data pair to obtain a preliminary training network, and then the network model is optimized by the second training data pair, and it is fine-tuned to obtain a trained generative adversarial network. Among them, the model uses the Adam optimizer and the batch size is set to 12. The generator network was trained with a learning rate of 2x10e-4 for 1000k iterations, and fine-tuned with a learning rate of 1x10e-4 for 40k iterations. An exponential moving average (EMA) was used for more stable training. The model was trained using L1 loss, perceptual loss, and GAN loss with a weight ratio of 1, 1, and 0.1.

[0040] In addition, the peak signal-to-noise ratio (PSNR) can also be used as an evaluation indicator of the model. It is one of the standard indicators for evaluating the quality of reconstructed images. A higher PSNR indicates a higher image quality, which can be expressed as: Among them, MaxVal represents the maximum pixel value of the HR image, and MSE represents the mean square error.

[0041] S103, obtaining a target image to be reconstructed, dividing the target image into at least two sub-blocks, generating reconstruction results of each sub-block based on a trained generative adversarial network, and splicing the reconstruction results based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image.

[0042] In the embodiment of the present specification, after the generative adversarial network is trained, the satellite image to be reconstructed, i.e., the target image, will be obtained. Since the target image is large, it needs to be read in blocks first to obtain several sub-blocks with smaller areas. For example, it can be read in blocks of 500*500, and then the sub-blocks are respectively input into the trained generative adversarial network to obtain the reconstruction results of each sub-block. Since splicing lines are easily generated at the splicing of images, the reconstruction results will be spliced ​​by the edge gradient fusion method to obtain the reconstructed image that finally completes the super-resolution reconstruction.

[0043] In one possible implementation, the step of obtaining a target image to be reconstructed, dividing the target image into at least two sub-blocks, generating reconstruction results of each sub-block based on a trained generative adversarial network, and splicing the reconstruction results based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image includes: Acquire a target image to be reconstructed, and read the target image into at least two sub-blocks, where there is an overlapping area of ​​a preset number of pixels between adjacent sub-blocks; Generate a reconstruction result of each sub-block based on a trained generative adversarial network; For overlapping pixels corresponding to overlapping areas in the reconstruction results, the weighted pixel sum of edge distances of overlapping pixels at the same position in each reconstruction result is taken as the final pixel value of the position to obtain a reconstructed image corresponding to the target image.

[0044] In the embodiments of this specification, in order to achieve edge gradient fusion, an overlapping area with a certain number of pixels is set between the sub-blocks. For the pixels in the overlapping area, the final pixel value after splicing is the weighted pixel sum of the edge distances of the overlapping sub-blocks at that position. Specifically, taking the overlapping area with 150 pixels as an example, for two sub-images A and B with one side overlapping at the edge of the large image, the pixels in the overlapping area are It can be expressed as: Among them, ij represents the position of the pixel, , Respectively represent the distance from the pixel at this point to the edge of sub-images A and B. The closer to the edge of the overlapping area, the less the pixel value of the sub-image contributes to the final pixel value of the point.

[0045] For the case where there are overlapping areas between the four images in the middle of the large image, the pixel value of this point is the weighted sum of the pixel points corresponding to the four sub-images according to the distance. It can be expressed as: in, , , , Respectively represent the distance from the pixel to the edge of sub-image A, B, C, and D.

[0046] The following will be combined with the attached Figure 2 , the satellite image super-resolution reconstruction device based on drone image assistance provided by the embodiment of the present application is introduced in detail. It should be noted that the attached Figure 2 The satellite image super-resolution reconstruction device based on drone image assistance is used to implement the present application Figure 1For the convenience of explanation, only the part related to the embodiment of the present application is shown. For the specific technical details not disclosed, please refer to the present application. Figure 1 The embodiment shown.

[0047] See also Figure 2 , Figure 2 Schematic diagram of a satellite image super-resolution reconstruction device based on drone image assistance provided by an embodiment of the present application. Figure 2 As shown, the device comprises: The acquisition module 201 is used to acquire the aerial image collected by the drone, perform aerial triangulation based on the image control points set in the aerial image, and orthorectify the aerial image according to the digital elevation model generated by the measurement result to obtain a digital orthophoto map; A training module 202 is used to select first images of different terrain areas in the digital orthophoto map, obtain second images after downsampling the first images, construct first training data pairs based on the first images and the second images, and train a generative adversarial network based on the first training data pairs, wherein each of the first training data pairs includes a high-resolution image and a low-resolution image with the same area size and area coordinates; The reconstruction module 203 is used to obtain the target image to be reconstructed, read the target image into at least two sub-blocks, generate reconstruction results of each sub-block based on the trained generative adversarial network, and splice the reconstruction results based on the edge gradient fusion method to obtain the reconstructed image corresponding to the target image.

[0048] In one possible implementation, the acquisition module 201 is specifically used for: Acquire aerial images collected by a drone based on oblique photography, and display the aerial images; A setting instruction for the aerial image is received, image control points in the aerial image are determined, and aerial triangulation is performed based on each of the image control points to obtain measurement data.

[0049] In one possible implementation, the acquisition module 201 is further specifically used for: Projecting pixel coordinates in the aerial image into three-dimensional coordinates in a three-dimensional space according to the measurement results, determining three-dimensional point cloud data in the three-dimensional space, and constructing a digital elevation model according to the three-dimensional point cloud data; After optimizing each of the three-dimensional coordinates based on the ground elevation value of the digital elevation model, resampling the optimized three-dimensional coordinates to obtain new pixel coordinates, and generating a preliminary orthophoto map corresponding to each of the aerial images based on the new pixel coordinates; The preliminary orthophoto images are stitched together to obtain a digital orthophoto image.

[0050] In one embodiment, the training module 202 is specifically used for: After adding Gaussian noise to the second image, Gaussian blur processing is performed on the second image to obtain a third image; Cutting the first image and the third image into at least two first sub-images and a second sub-image respectively, wherein the first sub-image and the second sub-image have the same image size; The first sub-image and the second sub-image corresponding to the same image coordinates are constructed as a first training data pair, and the first training data pairs are integrated to obtain a training data set.

[0051] In one possible implementation, the training module 202 is further configured to: Acquire a regional satellite image of each of the terrain areas, and adjust the regional satellite image based on geometric correction so that the geographical location of the regional satellite image is consistent with the first image; constructing a second training data pair based on the first image and a regional satellite image; The first training data pairs and the second training data pairs are integrated to obtain a training data set.

[0052] In one possible implementation, the training module 202 is further configured to: Obtain an initial generative adversarial network, wherein the generator of the initial generative adversarial network is a deep network with at least two dense residual learning blocks, and the discriminator of the initial generative adversarial network is a U-Net discriminator with skip connections; The initial generative adversarial network is trained based on the first training data pair to obtain a preliminary training network, and the preliminary training network is optimized based on the second training data pair to obtain a trained generative adversarial network.

[0053] In one possible implementation, the reconstruction module 203 is specifically configured to: Acquire a target image to be reconstructed, and read the target image into at least two sub-blocks, where there is an overlapping area of ​​a preset number of pixels between adjacent sub-blocks; Generate a reconstruction result of each sub-block based on a trained generative adversarial network; For overlapping pixels corresponding to overlapping areas in the reconstruction results, the weighted pixel sum of edge distances of overlapping pixels at the same position in each reconstruction result is taken as the final pixel value of the position to obtain a reconstructed image corresponding to the target image.

[0054] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented with the help of software and / or hardware. The "unit" and "module" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a field programmable gate array (FPGA), an integrated circuit (IC), etc.

[0055] Each processing unit and / or module of the embodiments of the present application may be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or may be implemented by software that executes the functions described in the embodiments of the present application.

[0056] See also Figure 3 , which shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, the electronic device can be used to implement Figure 1 The method in the embodiment shown. Figure 3 As shown, the electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0057] The communication bus 302 is used to realize the connection and communication between these components.

[0058] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0059] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0060] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts of the entire electronic device 300, and executes various functions and processes data of the electronic device 300 by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics central processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0061] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally be at least one storage device located away from the aforementioned processor 301. As Figure 3 As shown, the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.

[0062] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the satellite image super-resolution reconstruction application based on drone image assistance stored in the memory 305, and specifically perform the following operations: Acquire aerial images collected by drones, perform aerial triangulation based on image control points set in the aerial images, and orthorectify the aerial images according to a digital elevation model generated by the measurement results to obtain a digital orthophoto map; Selecting first images of different terrain areas in the digital orthophoto map, downsampling the first images to obtain second images, constructing first training data pairs based on the first images and the second images, and training a generative adversarial network based on the first training data pairs, wherein each of the first training data pairs includes a high-resolution image and a low-resolution image with the same area size and area coordinates; Obtain a target image to be reconstructed, divide the target image into at least two sub-blocks, generate reconstruction results of each sub-block based on a trained generative adversarial network, and splice the reconstruction results based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image.

[0063] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the above method are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0064] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0065] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0066] In the several embodiments provided in the present application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0067] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0068] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0069] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0070] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct related hardware, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0071] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A satellite image super-resolution reconstruction method based on drone image assistance. It is characterized in that The method comprises: Acquire aerial images collected by drones, perform aerial triangulation based on image control points set in the aerial images, and orthorectify the aerial images according to a digital elevation model generated by the measurement results to obtain a digital orthophoto map; Selecting first images of different terrain areas in the digital orthophoto map, downsampling the first images to obtain second images, constructing first training data pairs based on the first images and the second images, and training a generative adversarial network based on the first training data pairs, wherein each of the first training data pairs includes a high-resolution image and a low-resolution image with the same area size and area coordinates; Obtain a target image to be reconstructed, divide the target image into at least two sub-blocks, generate reconstruction results of each sub-block based on a trained generative adversarial network, and splice the reconstruction results based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image.

2. The method according to claim 1, It is characterized in that The step of acquiring the aerial image collected by the drone and performing aerial triangulation based on the image control points set in the aerial image includes: Acquire aerial images collected by a drone based on oblique photography, and display the aerial images; A setting instruction for the aerial image is received, image control points in the aerial image are determined, and aerial triangulation is performed based on each of the image control points to obtain measurement data.

3. The method according to claim 1, It is characterized in that The digital elevation model generated according to the measurement result is used to orthorectify the aerial image to obtain a digital orthophoto map, including: Projecting pixel coordinates in the aerial image into three-dimensional coordinates in a three-dimensional space according to the measurement results, determining three-dimensional point cloud data in the three-dimensional space, and constructing a digital elevation model according to the three-dimensional point cloud data; After optimizing each of the three-dimensional coordinates based on the ground elevation value of the digital elevation model, resampling the optimized three-dimensional coordinates to obtain new pixel coordinates, and generating a preliminary orthophoto map corresponding to each of the aerial images based on the new pixel coordinates; The preliminary orthophoto images are stitched together to obtain a digital orthophoto image.

4. The method according to claim 1, It is characterized in that The step of constructing a first training data pair based on the first image and the second image comprises: After adding Gaussian noise to the second image, Gaussian blur processing is performed on the second image to obtain a third image; Cutting the first image and the third image into at least two first sub-images and a second sub-image respectively, wherein the first sub-image and the second sub-image have the same image size; The first sub-image and the second sub-image corresponding to the same image coordinates are constructed as a first training data pair, and the first training data pairs are integrated to obtain a training data set.

5. The method according to claim 4, It is characterized in that The step of integrating the first training data pairs to obtain a training data set includes: Acquire a regional satellite image of each of the terrain areas, and adjust the regional satellite image based on geometric correction so that the geographical location of the regional satellite image is consistent with the first image; constructing a second training data pair based on the first image and a regional satellite image; The first training data pairs and the second training data pairs are integrated to obtain a training data set.

6. The method according to claim 5, It is characterized in that The training of a generative adversarial network based on the first training data comprises: Obtain an initial generative adversarial network, wherein the generator of the initial generative adversarial network is a deep network with at least two dense residual learning blocks, and the discriminator of the initial generative adversarial network is a U-Net discriminator with skip connections; The initial generative adversarial network is trained based on the first training data pair to obtain a preliminary training network, and the preliminary training network is optimized based on the second training data pair to obtain a trained generative adversarial network.

7. The method according to claim 1, It is characterized in that The step of obtaining a target image to be reconstructed, dividing the target image into at least two sub-blocks, generating reconstruction results of each sub-block based on a trained generative adversarial network, and splicing each reconstruction result based on an edge gradient fusion method to obtain a reconstructed image corresponding to the target image includes: Acquire a target image to be reconstructed, and read the target image into at least two sub-blocks, where there is an overlapping area of ​​a preset number of pixels between adjacent sub-blocks; Generate reconstruction results of each sub-block based on the trained generative adversarial network; For overlapping pixels corresponding to overlapping areas in the reconstruction results, the weighted pixel sum of edge distances of overlapping pixels at the same position in each reconstruction result is taken as the final pixel value of the position to obtain a reconstructed image corresponding to the target image.

8. A satellite image super-resolution reconstruction device based on drone image assistance. It is characterized in that The device comprises: An acquisition module is used to acquire aerial images collected by a drone, perform aerial triangulation based on image control points set in the aerial images, and orthorectify the aerial images according to a digital elevation model generated by the measurement results to obtain a digital orthophoto map; A training module, configured to select first images of different terrain areas in the digital orthophoto map, obtain second images after downsampling the first images, construct first training data pairs based on the first images and the second images, and train a generative adversarial network based on the first training data pairs, wherein each of the first training data pairs includes a high-resolution image and a low-resolution image with the same area size and area coordinates; The reconstruction module is used to obtain the target image to be reconstructed, read the target image into at least two sub-blocks, generate reconstruction results of each sub-block based on the trained generative adversarial network, and splice the reconstruction results based on the edge gradient fusion method to obtain the reconstructed image corresponding to the target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer-readable storage medium has instructions stored therein, and when the instructions are executed on a computer or a processor, the computer or the processor executes the steps of the method according to any one of claims 1 to 7.

Images