Nighttime automatic focusing method, device and equipment for drone photography

Abstract

The present invention relates to a nighttime automatic focusing method, comprising: measuring the distance between a camera and a target to obtain target distance information; searching for an initial focal length position corresponding to the target distance information in a preset distance-focal length curve; photographing the target according to a number of different focal lengths near the initial focal length, thereby obtaining a number of target images; performing image denoising and enhancement processing on the number of target images based on a student network to obtain a number of denoised images; calculating the clarity of each denoised image by a clarity calculation function; searching for the corresponding focal length according to the maximum clarity obtained by the clarity calculation function to obtain an optimal focal length. The nighttime automatic focusing method of the present invention has the advantages of reducing the environmental impact when the shooting device is at night, and achieving the advantages of achieving fast and accurate focusing even in a nighttime environment or a low-light environment.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a nighttime automatic focusing method, device and equipment for unmanned aerial vehicle photography. Background Art

[0002] The auto-focus method in the prior art obtains the initial focal length according to the distance between the camera and the target, adjusts the focal length based on the initial focal length, and uses the clarity calculation function or contrast to calculate the image gradient corresponding to the adjusted focal length to find the focal length corresponding to the maximum clarity, thereby completing the auto-focusing. The clarity of the image is obtained by calculating the gradient change value of the edge of the object in the image, that is, the larger the gradient value of the edge of the object contour, the clearer the image, and the smaller the gradient value, the blurrier the image.

[0003] In the scene where the drone is working at night, due to insufficient ambient light, the images taken by the drone contain a lot of noise, and the noise and the physical contour edge are both high-frequency signals, which leads to high gradient values ​​nearby. Therefore, when the gradient-based focusing method is applied to the scene where the drone is working at night, it is easy to cause the search for the maximum clarity to often find the local maximum value, or the change of its clarity value does not match the result observed by the human eye, and it is impossible to find the best focus position smoothly. Summary of the invention

[0004] Based on this, the object of the present invention is to provide a nighttime automatic focusing method for drone photography.

[0005] A nighttime automatic focusing method for drone photography comprises the following steps:

[0006] S1: Measure the distance between the camera and the target to obtain the target distance information;

[0007] S2: searching for an initial focal length position corresponding to the target distance information in a preset distance-focal length curve;

[0008] S3: photographing the target according to a plurality of different focal lengths near the initial focal length, thereby obtaining a plurality of target images;

[0009] S4: performing image denoising and enhancement processing on the target images to obtain a number of denoised images;

[0010] S5: Calculate the clarity of each denoised image through a clarity calculation function;

[0011] S6: According to the maximum clarity obtained by the clarity calculation function, find the corresponding focal length to obtain the optimal focal length.

[0012] The nighttime automatic focusing method for drone photography described in the present invention obtains an initial focal length through laser ranging, and the initial focal length is used as the initial anchor point for focusing. According to experience, the optimal focal length at the same distance is not fixed, but the optimal focal length must be near the initial focal length. Therefore, after obtaining the initial focal length, the image clarity corresponding to the focal length near the initial focal length is calculated to find the optimal focal length. At the same time, in order to reduce the impact of the nighttime environment on the image clarity calculation caused by the shooting equipment, the present invention solves the problem of inaccurate clarity calculation caused by image noise generated at night by denoising the noise generated by the image at night in real time. Finally, fast and accurate focusing is achieved even in nighttime or low-light environments.

[0013] Furthermore, the plurality of different focal lengths in step S3 are selected to be focal lengths within ±10% of the initial focal length and within a variation of 2%;

[0014] Among them, the parameter limitation is the focal length at which the inventor cannot distinguish the change of the clarity value corresponding to the focal length with the naked eye after multiple tests. By using the image corresponding to the focal length as the target image to calculate the clarity, the theoretically optimal focusing position can be found.

[0015] Furthermore, the clarity calculation function described in step S6 is a Sobel-based clarity calculation function, and the Sobel-based clarity calculation function is expressed as:

[0016]

[0017] Where M and N are the resolutions of the image. is the Sobel gradient value, expressed as:

[0018]

[0019] in, and Calculated by the following formula:

[0020]

[0021]

[0022]

[0023]

[0024] Among them, f(x,y) is the gray value of the image pixel, which is expressed by the matrix as:

[0025]

[0026]

[0027]

[0028]

[0029] Among them, A is the denoised image.

[0030] The Sobel-based clarity calculation function fully considers the gradient transformation in the horizontal, vertical and 45° directions, thereby obtaining the most accurate clarity value.

[0031] Furthermore, the denoising and enhancement described in step S4 are performed by a student network;

[0032] The trainable parameters of the student network are trained by using the corresponding camera noisy images and the denoised images output by the teacher network as training labels in the student network training data set:

[0033] De-noising the noisy image of the camera through the student network to obtain a denoised image output by the student network;

[0034] According to the denoised image output by the student network and the denoised image output by the teacher network;

[0035] Through the total loss, the parameters of the student network are adjusted and updated;

[0036] The student network training data set is constructed by the following steps:

[0037] Get the noisy image from the camera;

[0038] De-noising the camera noisy image through the trained teacher network to obtain a denoised image output by the teacher network;

[0039] The denoised image output by the teacher network and the corresponding camera noisy image are used to form the student network training data;

[0040] Among them, training the student network with noisy images from the camera allows the student network to learn the noise distribution of the corresponding camera device, thereby improving the denoising effect of the student network. At the same time, using the denoised images output by the teacher network as supervision information, the student network can learn the functions and characteristics of the teacher network.

[0041] The trainable parameters of the teacher network are trained by using the blurred noisy images and clear images in the teacher network training data set:

[0042] De-noising the blurred noisy image through the teacher network to obtain the de-noised image output by the teacher network;

[0043] The total loss is calculated based on the denoised image and the clear image output by the teacher network;

[0044] Through the total loss, the parameters of the teacher network are adjusted and updated;

[0045] The teacher network training data set is obtained by the following steps:

[0046] Get clear images;

[0047] Randomly add blur kernel, Gaussian noise and / or Poisson noise to the clear image to form a blurred noisy image;

[0048] The clear image and its corresponding blurred and noisy image form the teacher network training data.

[0049] Among them, the teacher network is more complex and more versatile than the student network. Therefore, complex data sets can enable the teacher network to learn more functions. Here, blur kernel, Gaussian noise and / or Poisson noise are used in the teacher network to add noise to the clear image. Therefore, the trained teacher network will have the functions of deblurring and denoising, and the student network will learn these functions.

[0050] Furthermore, the total loss can be expressed as:

[0051] L total =L 1 +L percep +0.1*L G

[0052] Among them, L 1 is the absolute value error, L percep is the perceptual loss, L G To identify the loss;

[0053] The absolute value error can be expressed as:

[0054] L 1 =mean||G(x i )-y i ||

[0055] Among them, G(x i ) is the output of the network to be trained, y i is the expected image, the output of the network to be trained is the denoised image output by the training subject; the expected image is the supervision image;

[0056] The perceptual loss can be expressed as:

[0057]

[0058] in, is the feature map extracted by the network to be trained, Feature map extracted for the desired image;

[0059] The discrimination loss can be expressed as:

[0060]

[0061] Among them, x i is the output of the network to be trained, It is the probability distribution of whether it is true or not determined by U-Net;

[0062] The discrimination loss uses U-Net to determine whether a single pixel of the denoised image output by the training network is real.

[0063] Among them, the absolute value error focuses more on PSNR (Peak Signal to Noise Ratio) to evaluate the image quality index, so that the network to be trained can adjust the parameters by generating image quality. Secondly, the perceptual loss calculates the difference between the extracted features of the output of the network to be trained and the expected image in the feature extraction process, so that the network to be trained can adjust the parameters from the perspective of the whole and features. Finally, the discriminative loss uses the difference between the output of the network to be trained and a single pixel of the expected image to adjust the parameters from the perspective of the pixel level.

[0064] In addition, the weights of the three loss functions are divided into 1.0, 1.0, and 0.1. The mean absolute error and perceptual loss enable the network to globally consider the difference between the output of the network to be trained and the expected image, so that the network can learn good denoising capabilities. The discrimination loss observes from a pixel-level perspective, so that the network to be trained can judge the difference more finely, and then the network to be trained can acquire image enhancement capabilities after learning. At the same time, since the discrimination loss calculates the difference for a single pixel, it is easy to overfit. Here, by using a weight of 0.1, it is ensured that the network will not overfit during the training process.

[0065] A nighttime automatic focusing device is used to execute any of the above-mentioned nighttime automatic focusing methods for drone photography.

[0066] A nighttime focusing device for a drone includes a distance measuring device, a camera, and the above-mentioned nighttime automatic focusing device;

[0067] The distance measuring device is used to measure the distance between the camera and the target, obtain the target distance information, and input the information into the nighttime automatic focusing device;

[0068] The camera adjusts the focal length according to the signal of the nighttime automatic focusing device to capture a plurality of target images, and inputs the plurality of target images into the nighttime automatic focusing device;

[0069] After processing the input data, the nighttime automatic focusing device outputs a signal to adjust the camera to the optimal focal length.

[0070] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a schematic diagram of the structure of a night focus device;

[0072] Figure 2 The flowchart of the night auto focus method is as follows;

[0073] Figure 3 Flowchart of the training method for the teacher network;

[0074] Figure 4 Flowchart of the training method for the student network. DETAILED DESCRIPTION

[0075] The present invention combines a denoising algorithm with a distance- and gradient-based focusing method to minimize the impact of nighttime and low-light environments, thereby forming a nighttime automatic focusing method for drone photography, and based on this method, forming a drone nighttime focusing device.

[0076] Please also see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram of the structure of the night focus device. Figure 2 The figure is a flow chart of the night autofocus method.

[0077] The drone night focus device comprises a distance measuring device 1, a camera 2, and a night auto focus device 3. The night auto focus device 3 comprises an initial focal length search module 31, an image denoising module 32, a clarity calculation module 33, and an optimal focal length search module 34.

[0078] The distance measuring device 1 is used to execute step S1: measuring the distance between the camera and the target to obtain target distance information.

[0079] In one embodiment, the distance measuring device is a laser rangefinder, which emits a laser beam to the target. After the target reflects the laser beam, the laser rangefinder receives the reflected laser beam, thereby calculating parameters such as time difference and measuring the distance between the camera and the target.

[0080] The initial focal length search module 31 is used to execute step S2: searching for an initial focal length position corresponding to target distance information in a preset distance-focal length curve.

[0081] Specifically, the preset distance-focal length curve is a curve of different focal lengths corresponding to different camera distances, which is obtained by artificial fitting. Accordingly, after obtaining the target distance information, a relatively optimal initial focal length can be obtained through the distance-focal length curve.

[0082] The distance measuring device measures the distance between the camera and the current target to calibrate the approximate distance between the camera and the target, i.e., the initial anchor point. According to the inventor's experience, at the same distance, the optimal focal length is not fixed. Not only will there be a big difference between day and night, but the optimal focal length will also be affected by the environment such as temperature and humidity, so that the relatively optimal initial anchor point obtained may not be the optimal focal length observed by the naked eye, but the optimal focal length must be near the initial anchor point. Therefore, after obtaining the initial anchor point, the optimal focal length can be found near it, thereby improving the efficiency of focusing.

[0083] The camera 2 is used to execute step S3: photographing the target according to a plurality of different focal lengths near the initial focal length, thereby obtaining a plurality of target images.

[0084] Preferably, the several different focal lengths are selected as the target focal lengths within ±10% of the initial focal length and within a variation of 2%. The parameter definition is the focal length parameter that the inventors cannot distinguish the change of the clarity value corresponding to the focal length with the naked eye after multiple tests.

[0085] The image denoising module 32 is used to execute step S4: performing image denoising and enhancement processing on the plurality of target images to obtain a plurality of denoised images.

[0086] The clarity calculation module 33 is used to execute step S5: calculating the clarity of each denoised image by using a clarity calculation function.

[0087] Specifically, the clarity calculation function is a Sobel operator, which is expressed as:

[0088]

[0089] in, A is the denoised image.

[0090] In one embodiment, the clarity of each denoised image is calculated by using a Sobel-based clarity calculation function, wherein the Sobel-based clarity calculation function is:

[0091]

[0092] Where M and N are the resolutions of the image. is the Sobel gradient value, expressed as:

[0093]

[0094] in, and Calculated by the following formula:

[0095]

[0096]

[0097]

[0098]

[0099] Among them, f(x,y) is the gray value of the image pixel, and is expressed by the matrix as:

[0100]

[0101]

[0102]

[0103]

[0104] Among them, A is the denoised image.

[0105] At the same time, the Sobel-based clarity fully considers the gradient changes in the horizontal, vertical and 45° directions, further reducing the noise impact of the shooting equipment at night or in low-light environments.

[0106] The optimal focal length search module 34 is used to execute step S6: identifying the maximum clarity calculated by the clarity calculation module 33, searching for the corresponding focal length, and obtaining the optimal focal length.

[0107] The maximum clarity indicates that the gradient value of the current image is the maximum value, that is, the image contour of the target edge is relatively clearest. The optimal focal length value can be determined by the maximum clarity, so that the camera focus is locked at the optimal focal length to ensure the accuracy of the focus.

[0108] The working principle of the nighttime automatic focusing device is as follows: by determining the distance between the camera and the target, an initial focal length corresponding to the target is obtained. Therefore, based on the initial focal length, a nearby focal length is obtained as an alternative focal length group, and a plurality of images of the corresponding focal length of the shooting device are obtained based on the alternative focal length group. By denoising the plurality of images, the edge gradient of the image is enhanced, the clarity is calculated, and the optimal focal length corresponding to the highest clarity among the plurality of clarity is identified, so that the camera focus is adjusted to lock to the optimal focal length, and the accuracy of focusing is ensured.

[0109] In order to improve the accuracy of finding the best focus, the existing technology removes the noise in the image through a denoising model. Generally, the denoising model extracts (convolves) the noise features of the target image multiple times, performs inverse feature extraction (deconvolution, inverse transformation, upsampling) on ​​the noise features, and performs image reconstruction with the residual operation to achieve denoising. Alternatively, a noisy image is input, and a feature map is extracted multiple times and the loss is calculated with the noise-free image. Finally, the feature map is adjusted to output an accurate denoised image, and the clarity is calculated based on the denoised image. The optimal focal length position is determined based on the maximum clarity value. However, in fact, the denoising operation is prone to cause image blur.

[0110] In order to improve the focusing accuracy of drones at night, it is necessary to maintain real-time denoising of the captured images, while ensuring that the target edges of the denoised images will not be blurred due to the denoising process. Therefore, a model with image enhancement and denoising functions is required, but the models with the above functions in the prior art are basically large denoising models. However, the existing large denoising models require high computing power support to achieve real-time denoising and enhancement of captured images, and the power consumption of high computing power chips will reduce the endurance of drones.

[0111] However, the existing lightweight denoising model has a small structure (low depth), which results in insufficient semantic extraction depth during the training process and cannot meet the requirements of real-time and accurate image denoising and enhancement when the drone is focusing at night.

[0112] In order to make the lightweight model have the functions and performance of the large model, the present invention further studies compressing the large model into the lightweight model through the knowledge distillation framework, so that the lightweight model has some functions of the large model.

[0113] The knowledge distillation framework inputs the same training set data as the student network to be trained (lightweight model) through the trained teacher network (large model). The trained teacher network outputs the processed images in the training set data as the labels (soft labels) of the student network, guiding the student network to learn the functions of the teacher network, so that the student network can quickly acquire the knowledge of the teacher network.

[0114] In addition, due to the small structure (low depth) of the student network, the semantic extraction depth during the training process is insufficient, which affects the training efficiency and even causes the student network to be unable to fully learn the functions of the teacher network. In order to improve the training efficiency and the accuracy of the student network, the present invention further envisions that the student network learns the functions of the teacher network from multiple perspectives, including the overall output of the teacher network, the semantic features of each layer of the teacher network output, and each pixel of the teacher network output, so that the student network can fully learn the functions of the teacher network.

[0115] Specifically, in order to optimize the training effect of the student network and fully learn the functions of the teacher network, the teacher network and the student network select the same type of model, such as the same type of VGG, ResNet or GoogLeNet and other networks with image feature extraction, which is not specifically limited in this application.

[0116] Among them, the teacher network has a larger structure and deeper depth than the student network, and has more layers for feature extraction, feature processing, etc. Therefore, compared with the student network, the teacher network can learn a larger distribution of data, making the teacher network highly versatile, accurate, and robust.

[0117] In this embodiment, the teacher network uses 20 layers of convolutional layers as feature extraction, and the number of channels of the feature map is transformed through the pooling layer between the convolutional layers, and then the image is reconstructed through 10 layers of deconvolution (inverse transformation, upsampling, etc.) operations with the residual structure, and finally the denoising result is output through the fully connected layer. The student network uses 10 layers of convolutional layers as feature extraction, and the number of channels is changed through the pooling layer between the convolutional layers, and then the image is reconstructed through 5 layers of deconvolution (inverse transformation, upsampling, etc.) operations with the residual structure, and finally the denoising result is output through the fully connected layer.

[0118] Specifically, in order to enable the teacher network to have denoising and image enhancement functions, the teacher network training dataset is constructed through the following steps:

[0119] D101: Get clear images.

[0120] Specifically, the visible light image with a resolution of 1920*1080 is cut into a target area image with a resolution of 640*360, i.e., a clear image. Since the processing speed is slow when the visible light image is denoised, and the focusing algorithm only focuses on the area of ​​interest, i.e., the target area, the denoising algorithm only needs to perform denoising on the target area image.

[0121] D102: Randomly add blur kernel, Gaussian noise and / or Poisson noise to the clear image to form a blurred noisy image.

[0122] Specifically, Gaussian noise and Poisson noise are randomly added to the clear image to simulate the noise of the camera itself, forming a small-resolution simulated noise image.

[0123] Next, the low-resolution simulated noise image is blurred by adding a blur kernel to obtain a blurred noisy image. Since the image is easily blurred during the image denoising and reconstruction process, which affects the subsequent focus algorithm judgment, adding a blur kernel can enable the model to have both denoising and deblurring functions.

[0124] D103: Combine the clear image and its corresponding blurred noisy image to form the teacher network training data.

[0125] Specifically, by performing steps D101 and D102 multiple times, multiple groups of clear images and corresponding blurred and noisy images are obtained to form a teacher network training data set. In this embodiment, the teacher network training data set, the teacher network test data set, and the teacher network verification data set can be divided into 8:1:1 ratios.

[0126] See also Figure 3 , Figure 3 Flowchart of the teacher network training method.

[0127] After obtaining the teacher network training data set, the teacher network can be trained by using the corresponding fuzzy noisy images and clear images in the teacher network training data set, and the steps are as follows:

[0128] A101: De-noise the blurred noisy image through the teacher network to obtain the denoised image output by the teacher network.

[0129] A102: Calculate the total loss based on the denoised image and clear image output by the teacher network.

[0130] The total loss includes absolute value error, perceptual loss and discrimination loss.

[0131] Specifically, the absolute value error is obtained in the following manner.

[0132] The absolute value error between the denoised image output by the teacher network and the clear image can be expressed as:

[0133] L 1 =mean||G(x i )-y i ||

[0134] Among them, G(x i ) is a clear image in the teacher network training dataset, y i Denoised image output by the teacher network.

[0135] The perceptual loss is calculated in the following manner.

[0136] The denoised image and the clear image of the teacher network are processed by the pre-trained VGG19 network to obtain the feature images output by the convolution layer before each pooling layer in the VGG19 network, which are 5 groups of feature images. Then, the absolute value error of the feature images corresponding to the denoised image and the clear image is calculated. The perceptual loss can be expressed as:

[0137]

[0138] Among them, the perceptual loss is mainly calculated by comparing the feature maps extracted from the clear image and the denoising features extracted from the teacher network, that is, the loss of high-level information, that is, the content and global structure, so that the model can focus on the differences between image features instead of only calculating the loss of the final output and the clear image.

[0139] The VGG19 network structure includes 16 convolutional layers and 3 fully connected layers, where the convolutional layers are divided into five groups, which can be expressed as [2,2,4,4,4]. After each group of outputs, a maximum pooling operation is performed to remove invalid features of the feature map and change the number of feature map channels.

[0140] The discrimination loss is calculated as follows:

[0141] The trained U-Net is used as the discriminant model, and the output of the teacher network is processed by the discriminant model. According to the probability distribution y of all 1s r The binary cross entropy loss is calculated, based on which the discrimination loss can be expressed as:

[0142]

[0143] Introducing U-Net as a discriminant model allows the teacher network to perform difference calculations on individual pixels generated by the teacher network from a pixel perspective during training, taking into account the local features of the image, and ultimately making the output of the teacher network closer to a clear image.

[0144] The trained U-Net is obtained by the following steps:

[0145] The output of the teacher network and the clear image are input into the U-Net identification model to be trained, and the output is a probability distribution y of all 1s r , or the probability distribution y of all zeros f , by calculating the clear image corresponding to y r , and the teacher network output corresponds to y f The binary cross entropy loss is expressed as:

[0146]

[0147]

[0148] When the U-Net identification model identifies it as true, the input is a clear image, and When the U-Net identification model identifies a false positive, the input is the output of the teacher network. After calculating the binary cross entropy loss multiple times and adjusting the U-Net identification model parameters according to the loss value, the trained U-Net is obtained after the preset loss value is finally reached.

[0149] Preferably, the weights of the three loss functions are 1.0, 1.0, and 0.1, and the total loss can be expressed as:

[0150] L total =L 1 +L percep +0.1*L G

[0151] Among them, since the identification model will perform difference calculations on a single pixel, it is prone to overfitting. Here, by using a weight of 0.1, it is ensured that the network will not overfit during the training process.

[0152] A103: Adjust and update the parameters of the teacher network through the total loss.

[0153] It can be understood that in this embodiment, for the teacher network training method, steps A101 to A103 can be repeatedly performed multiple times to train the teacher network through the teacher network training data set, the teacher network test data set and the teacher network verification data set, until the total loss value calculated according to the teacher network test data set and the teacher network verification data set is within a preset range, or the number of training times reaches a preset threshold, and finally the trained teacher network is obtained. After the training is completed, the teacher network inputs a noise map to accurately denoise and enhance the noise map, and outputs a denoised image.

[0154] Among them, if the student network directly adopts the data set of the teacher network, the result of the student network processing the real noisy image will be blurred. This is because the data distribution range of random noise is wide, and the student network has a low depth and cannot cover all of it. At the same time, it is difficult to obtain matching real clear images and real noisy images. Therefore, in order to allow the student network to learn the noise distribution of a specific device in a targeted manner, the present invention adopts real noisy images for application scenarios, and trains the student network based on the denoising of the real noise image by the trained teacher network to obtain a relatively real and clear image.

[0155] Specifically, in order to enable the student network to learn the denoising task of the corresponding device in a targeted manner and reduce the training data, the following steps are used to build the student network training data set:

[0156] D201: Get the noisy image from the camera.

[0157] Specifically, the camera noisy image is a real noisy image taken at night, and has a resolution of 640*360.

[0158] D202: De-noising the noisy image of the camera through the trained teacher network to obtain a denoised image output by the teacher network.

[0159] D203: The denoised image output by the teacher network and the corresponding camera noisy image are combined to form the student network training data.

[0160] Specifically, by performing steps D201 and D202 multiple times, multiple groups of denoised images output by the teacher network and corresponding camera noisy images are obtained to form a training data set. In this embodiment, the student network training data set, the student network test data set, and the student network verification data set can be divided into an 8:1:1 ratio.

[0161] See also Figure 4 , Figure 4 Flowchart of the student network training method.

[0162] After obtaining the student network training data set, the student network can be trained by using the corresponding camera noisy images in the training data set and the denoised images output by the teacher network as training labels. The training steps are as follows:

[0163] A201: De-noise the noisy image from the camera through the student network to obtain the denoised image output by the student network.

[0164] A202: Calculate the total loss based on the denoised image output by the student network and the denoised image output by the teacher network.

[0165] The total loss includes absolute value error, perceptual loss and discrimination loss.

[0166] A203: Adjust and update the parameters of the student network through the total loss.

[0167] The above loss calculation is specifically the same as the teacher network training step, wherein the denoised image output by the teacher network in steps A202~A203 can be represented as the clear image in the teacher network training step, and the denoised image output by the student network in steps A202~A203 can be represented as the denoised image in the teacher network training step, which will not be repeated here.

[0168] It can be understood that, in this embodiment, for the student network training method, the student network can be trained by repeatedly performing steps A201 to A205 through the student network training data set, the student network test data set and the student network verification data set, until the total loss value calculated according to the student network test data set and the student network verification data set is within a preset range, or the number of training times reaches a preset threshold, and finally the trained student network is obtained. After the training is completed, the student network inputs a noise map to accurately denoise and enhance the noise map, and outputs a denoised image.

[0169] Compared with the prior art, the present invention uses a lightweight noise reduction model to ensure that the shooting device is not affected by night noise when automatically focusing at night. At the same time, through the improved clarity calculation function based on Sobel, the gradients of the horizontal and vertical axes and the 45° angle direction are fully considered, so that the night clarity calculation is more accurate, and the accurate optimal focal length is obtained.

[0170] In addition, in order to enable resource-constrained drones to have real-time denoising with high accuracy, the present invention uses a knowledge distillation framework to allow lightweight denoising models to have the functions and performance of large models. In particular, by adding a blur kernel to the training set of a large model, the lightweight denoising model can have a deblurring function. At the same time, in order to improve the denoising performance, the denoising ability of the lightweight denoising model to learn the large model is improved through the mean absolute error and perceptual loss. In order to enable the model to have the image enhancement function while denoising, the model is trained from a single pixel perspective through the identification model, so that the model greatly improves the image enhancement ability. In addition, due to the different learning capabilities of large models and lightweight models, training data sets are designed separately to make the lightweight model more sensitive to the noise generated by a certain device, thereby achieving the best denoising effect.

[0171] Based on the same inventive concept, the present application also provides an electronic device, which may be a terminal device such as a server, a desktop computing device or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute a program to implement the nighttime automatic focusing method for drone photography of an embodiment of the present invention; and the memory is used to store a computer program executable by the processor.

[0172] Based on the same inventive concept, the present application also provides a computer-readable storage medium, corresponding to the embodiment of the aforementioned night autofocus method, wherein the computer-readable storage medium stores a computer program thereon, and when the program is executed by the processor, the steps of the night autofocus method for drone photography recorded in any of the above embodiments are implemented.

[0173] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0174] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A nighttime automatic focusing method for drone photography, characterized in that: The following steps are involved: S1: Measure the distance between the camera and the target to obtain the target distance information; S2: searching for an initial focal length position corresponding to the target distance information in a preset distance-focal length curve; S3: photographing the target according to a plurality of different focal lengths near the initial focal length, thereby obtaining a plurality of target images; S4: performing image denoising and enhancement processing on the target images to obtain a number of denoised images; The image denoising and enhancement are performed by a student network; the training parameters of the student network are trained by using the corresponding camera noisy images in the training data set of the student network and the denoised images output by the teacher network as training labels: De-noising the noisy image of the camera through the student network to obtain a denoised image output by the student network; According to the denoised image output by the student network and the denoised image output by the teacher network; Through the total loss, the parameters of the student network are adjusted and updated; The student network training data set is constructed by the following steps: Get the noisy image from the camera; De-noising the camera noisy image through the trained teacher network to obtain a denoised image output by the teacher network; The denoised image output by the teacher network and the corresponding camera noisy image are used to form the student network training data; The training parameters of the teacher network are trained by using the blurred noisy images and clear images in the teacher network training data set: De-noising the blurred noisy image through the teacher network to obtain the de-noised image output by the teacher network; The total loss is calculated based on the denoised image and the clear image output by the teacher network; Through the total loss, the parameters of the teacher network are adjusted and updated; The teacher network training data set is obtained by the following steps: Get clear images; Randomly add blur kernel, Gaussian noise and / or Poisson noise to the clear image to form a blurred noisy image; The clear image and its corresponding blurred and noisy image are used to form the teacher network training data; Among them, the total loss is: in, is the absolute value error, For perceived loss, To identify the loss; The absolute value error is: in, is the output of the network to be trained, is the expected image, the output of the network to be trained is the denoised image output by the training subject; the expected image is the supervision image; The perceptual loss is: in, is the feature map extracted by the network to be trained, Feature map extracted for the desired image; The identification loss is: in, is the output of the network to be trained, It is the probability distribution of whether it is true or not determined by U-Net; The discrimination loss uses U-Net to determine whether a single pixel of the denoised image output by the training network is real; S5: Calculate the clarity of each denoised image through a clarity calculation function; S6: Identify the maximum definition calculated in step S5, find its corresponding focal length, and obtain the optimal focal length.

2. The nighttime automatic focusing method for drone photography according to claim 1, characterized in that: The plurality of different focal lengths in step S3 are selected to be focal lengths within ±10% of the initial focal length and within a variation of 2%.

3. The nighttime automatic focusing method for drone photography according to claim 2, characterized in that: The definition calculation function described in step S6 is a Sobel-based definition calculation function, which is expressed as: Where M and N are the resolutions of the image. is the Sobel gradient value, expressed as: in, , , and Calculated by the following formula: in, is the gray value of the image pixel, which is expressed by the matrix: in, is the denoised image.

4. A nighttime automatic focusing device, characterized in that: It includes an initial focal length search module, an image denoising module, a clarity calculation module, and an optimal focal length search module; The initial focal length search module is used to search for the initial focal length position corresponding to the target distance information in a preset distance-focal length curve; wherein the target distance information is measured by a distance measuring device to measure the distance between the camera and the target; The image denoising module performs image denoising and enhancement processing on a plurality of target images to obtain a plurality of denoised images; wherein the plurality of target images are obtained by photographing the target at a plurality of different focal lengths near the initial focal length through a camera; wherein the image denoising and enhancement are performed through a student network; the training parameters of the student network are the corresponding camera noisy images in the student network training data set and the denoised images output by the teacher network as training labels, and the student network is trained: De-noising the noisy image of the camera through the student network to obtain a denoised image output by the student network; According to the denoised image output by the student network and the denoised image output by the teacher network; Through the total loss, the parameters of the student network are adjusted and updated; The student network training data set is constructed by the following steps: Get the noisy image from the camera; De-noising the camera noisy image through the trained teacher network to obtain a denoised image output by the teacher network; The denoised image output by the teacher network and the corresponding camera noisy image are used to form the student network training data; The training parameters of the teacher network are trained by using the blurred noisy images and clear images in the teacher network training data set: De-noising the blurred noisy image through the teacher network to obtain the de-noised image output by the teacher network; The total loss is calculated based on the denoised image and the clear image output by the teacher network; Through the total loss, the parameters of the teacher network are adjusted and updated; The teacher network training data set is obtained by the following steps: Get clear images; Randomly add blur kernel, Gaussian noise and / or Poisson noise to the clear image to form a blurred noisy image; The clear image and its corresponding blurred and noisy image are used to form the teacher network training data; Among them, the total loss is: in, is the absolute value error, For the perceived loss and To identify the loss; The absolute value error is: in, is the output of the network to be trained, is the expected image, the output of the network to be trained is the denoised image output by the training subject; the expected image is the supervision image; The perceptual loss is: in, is the feature map extracted by the network to be trained, Feature map extracted for the desired image; The identification loss is: in, is the output of the network to be trained, It is the probability distribution of whether it is true or not determined by U-Net; The discrimination loss uses U-Net to determine whether a single pixel of the denoised image output by the training network is real; The clarity calculation module calculates the clarity of each denoised image through a clarity calculation function; The optimal focal length search module is used to identify the maximum clarity calculated by the clarity calculation module, find the corresponding focal length, and obtain the optimal focal length.

5. A nighttime focusing device for a drone includes a distance measuring device and a camera, characterized in that: Also includes the nighttime automatic focusing device as claimed in claim 4; The distance measuring device is used to measure the distance between the camera and the target, obtain the target distance information, and input the target distance information into the nighttime automatic focusing device; The camera adjusts the focal length according to the signal of the nighttime automatic focusing device to capture a plurality of target images, and inputs the plurality of target images into the nighttime automatic focusing device; After processing the input data, the nighttime automatic focusing device outputs a signal to adjust the camera to the optimal focal length.

6. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the nighttime autofocus method for drone photography as described in any one of claims 1 to 3 is implemented.

Images