Optical remote sensing camera high-precision rapid focusing method based on deep learning

Abstract

The invention discloses an optical remote sensing camera high-precision rapid focusing method based on deep learning, and belongs to the field of photoelectric detection imaging. The problems that an existing push-broom optical remote sensing camera needs to shoot the same scene for manual subjective judgment, the focusing period is long, and the focusing precision is difficult to guarantee are solved. According to the method, a deblurring deep learning model trained by a large number of samples is established, imaging is performed at different focal lengths in a traversal mode, a focusing interval is continuously reduced based on similarity evaluation of a deblurred image and an original image, the optimal focal plane position is rapidly determined, and the focusing period is shortened while high precision of image focusing is ensured. According to the invention, the focusing period is greatly shortened while the high precision of the image is ensured. The invention relates to an automatic focusing method with universality. According to the method, a corresponding automatic focusing algorithm library does not need to be customized according to a lens, and the optimal focal length is selected based on image quality evaluation after traversal.

Description

Technical Field

[0001] The present invention belongs to the field of photoelectric detection imaging, and in particular relates to a high-precision and rapid focusing method for an optical remote sensing camera based on deep learning. Background Art

[0002] Focusing of remote sensing cameras is an important method to improve image quality. The imaging principle of a camera is based on the principle of pinhole imaging. When light passes through the optical center of the lens, it is projected onto the imaging photosensitive device, thereby capturing an image. Focusing is done by moving the entire lens or certain lenses in the lens to adjust the distance between the imaging plane and the lens so that the distance from the imaging plane to the optical center is equal to the image distance, and the object can be clearly imaged on the photosensitive device.

[0003] At present, traditional focusing requires obtaining multiple frames of images of the same scene to calculate the statistical characteristics of the image, and calculating the lens movement position based on the focus algorithm library during the corresponding period. On the one hand, this type of method is not universal and requires the customization of the corresponding algorithm library according to different lenses. On the other hand, it is difficult for mobile remote sensing cameras, especially push-broom remote sensing cameras, to obtain multiple frames of images of the same scene. The above reasons have led to the widespread use of manual focusing for remote sensing cameras, and the imaging scene must be consistent, which greatly lengthens the focusing cycle, and the accuracy is difficult to guarantee through manual judgment.

[0004] The present invention provides a high-precision and fast focusing method for an optical remote sensing camera based on deep learning. The method establishes a deblurring deep learning model trained with a large number of samples, uses a traversal method to image at different focal lengths, continuously narrows the focusing interval based on the similarity evaluation between the deblurred image and the original image, and finally determines the optimal focal plane position, which greatly shortens the focusing cycle while ensuring high image precision. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention proposes a high-precision and rapid focusing method for an optical remote sensing camera based on deep learning.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] A high-precision and fast focusing method for optical remote sensing cameras based on deep learning. The processing flow is as follows: Figure 1 The following steps are involved:

[0008] S1: Deploy the deblurring deep learning model N after training with a large number of samples;

[0009] Deploy the trained deblurring deep learning model N as Figure 2It is divided into three processes: dataset construction, model N construction, and model training. The dataset consists of A pairs of blurry and clear images. The blurry image is used as the input of model N, and the clear image is used as the label to measure the deblurring performance of the model. The deblurring model N directly inputs the blurry image M of size H×W and outputs the deblurred image M of size H×W. q The model training uses the loss function Loss to measure the degree of deblurring, and its expression is as follows: A ≥ 1000;

[0010]

[0011] Where D represents the grayscale level of image M, M t represents the clear image label, SSIM and MSE represent the peak-to-peak signal-to-noise ratio and mean square error respectively. The calculation methods of SSIM and MSE are as follows.

[0012]

[0013] Where a, b, c are constants, l, c, s represent brightness, contrast and structure respectively, and the calculation formula is as follows:

[0014]

[0015] where μ M represents the grayscale mean of the image, σ M represents the standard deviation of the image, Represents the covariance between images, c1, c2, c3 are constants.

[0016] S2: Determine the focusing range I of the remote sensing camera;

[0017] S3: Divide the focus range into n parts, that is, divide the focus range I into n ranges I1, ..., I2 with a step size of I / n. n ;

[0018] S4: Remote sensing camera with I1,…,I n The midpoint of the interval is the focal length. It is not necessary to shoot at the same location. Keep the sensor imaging posture consistent and image n times to obtain n images M1, ..., M with a size of H × W. n ;

[0019] S5: n images M1, ..., M n They are fed into the deblurring deep learning model N to obtain n corresponding deblurred images M of size H×W. q 1,…,M q n ;

[0020] S6: Calculate the similarity of n pairs of images in S5 respectively;

[0021] Imaged image M and deblurred image M q The similarity L is comprehensively measured by structural similarity (SSIM) and mean square error (MSE), and its expression is as follows:

[0022]

[0023] Where D is the image M and M q The similarity L is a monotonically increasing function. The larger the value of L, the closer the image M is to the deblurred image M. q The higher the similarity, the better the imaging quality of the current focal length.

[0024] S7: Select the focal length interval I with the highest similarity among n pairs of images x ;

[0025] S8: Determine whether to end the focusing.

[0026] There are two methods to judge the end of focusing. One is to manually set the threshold Y. If I x The similarity L x If it is greater than the threshold value Y, the focus adjustment is exited; if it is greater than the threshold value, the focus adjustment is ended; if it is less than the threshold value Y, I x As a new focusing interval I, steps S2 to S8 are repeated.

[0027] Another way is to record every time I x The similarity L x , that is [L x1 , …, L xn ], if after the n+1th iteration L xn+1 Greater than or equal to L xn The focal length of the nth time is taken as the best focal length and the focusing is exited. xn+1 Less than or equal to L xn Then the I of the n+1th round x As a new focusing interval I, steps S2 to S8 are repeated.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention uses a deep learning deblurring model to evaluate the focusing effect, which effectively overcomes the problem that push-broom optical remote sensing cameras are difficult to obtain multiple frames of images of the same scene and use manual focusing. While ensuring high image accuracy, the focusing cycle is greatly shortened.

[0030] 2. The high-precision and fast focusing method for optical remote sensing cameras based on deep learning provided in the present invention is a universal automatic focusing method. This method does not require the customization of the corresponding automatic focusing algorithm library according to the lens, but selects the optimal focal length based on the image quality evaluation after traversal. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the high-precision and fast focusing method of an optical remote sensing camera based on deep learning of the present invention;

[0032] Figure 2 It is a deobfuscated network deployment flow chart;

[0033] Figure 3 This is the U-Net deblurring network structure diagram

[0034] Figure 4 This is a schematic diagram of sensor imaging data selection DETAILED DESCRIPTION

[0035] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0036] A high-precision and fast focusing method for optical remote sensing cameras based on deep learning, the specific implementation process is as follows: Figure 1 The following steps are involved:

[0037] S1, deploys the deblurring deep learning model N after training with a large number of samples;

[0038] Deploy the deblurring deep learning model N trained with a large sample size. Figure 2 It is divided into three processes: data set construction, model N construction, and model training.

[0039] S1.1, Dataset Construction. One thousand clear remote sensing images with a size of 512×512 are selected, and one thousand blurred images with a corresponding size of 512×512 are obtained by downsampling, upsampling, Gaussian blurring, mean blurring, and adding other noise. The one thousand pairs of images are divided into training set, validation set, and test set in a ratio of 8:1:1.

[0040] S1.2 Defuzzification model N is constructed. Figure 3 As shown in Figure 1, the deblurring model uses U-Net, the input of the model is a blurred image of size 512×512, and the output is an image of size 512×512.

[0041] S1.3 Model training. Input the training set images into U-Net one by one. Assume that the input image is M and the model output image is M q , the deblurring degree is measured by the loss function Loss, and its expression is as follows.

[0042]

[0043] Where D represents the grayscale level of image M, M trepresents the clear image label, SSIM and MSE represent the peak-to-peak signal-to-noise ratio and mean square error respectively. The calculation methods of SSIM and MSE are as follows.

[0044]

[0045] Where a, b, c are constants, l, c, s represent brightness, contrast and structure respectively, and the calculation formula is as follows:

[0046]

[0047] where μ M represents the grayscale mean of the image, σ M represents the standard deviation of the image, Represents the covariance between images, c1, c2, c3 are constants.

[0048] The training process is an iterative process, and the model uses the loss back propagation to update the weights. During the training process, the validation set is used to evaluate the deblurring effect of the model, that is, to calculate the deblurred image M q With clear image M t The similarity L of . Similarity L is a monotonically increasing function, and the model weight with the highest similarity is retained. The calculation formula of similarity L is as follows.

[0049]

[0050] Where D represents the grayscale level of image M.

[0051] After the training is completed, the test set is used to evaluate the deblurring effect of the model, that is, to calculate the deblurred image M q With clear image M t If the deblurring effect of the model meets the focusing requirements, the model is deployed using the weight. Otherwise, the model learning rate, step size update strategy and other training parameters are adjusted and retrained.

[0052] S2, determine the focusing range I of the remote sensing camera;

[0053] Assume that the initial focusing range is [1μm, 11μm].

[0054] S3, dividing the focus range into n parts, that is, dividing the focus range I into n ranges I1, ..., I2 with a step size of I / n. n ;

[0055] Here n=10, that is, the focusing interval is divided into ten intervals of [1μm, 2μm], ..., [10μm, 11μm] with a step size of 1μm.

[0056] S4, remote sensing camera with I1,…,I nThe midpoint of the interval is the focal length. It is not necessary to shoot at the same location. Keep the sensor imaging posture consistent and image n times to obtain n images M1, ..., M with a size of H × W. n ;

[0057] The remote sensing camera maintains the imaging posture during the movement. It does not need to shoot the same location. It images 10 times with focal lengths of 1.5 μm, …, 10.5 μm, and obtains images M1, …, M2 with a size of 10240000×10240000. 10 .

[0058] S5: n images M1, ..., M n They are fed into the deblurring deep learning model N to obtain n corresponding deblurred images M of size H×W. q 1,…,M q n ;

[0059] Since images M1,…,M 10 The size is 10240000×10240000, which is much larger than the input size of the deblurring model U-Net, which is 1024×1024. Figure 4 As shown in the figure, nine 1024×1024 images are taken from the edge, middle and other nine directions of image M as input images, that is, [M n-1 , …, M n-9 ], each group of 9 images is input into the deblurring model U-Net to obtain the deblurred image set M q 1,…,M q 9, where M q n Contains 9 images, namely [M q n-1 , …, M q n-9 ].

[0060] S6: Calculate the similarity of n pairs of image sets in S5 respectively;

[0061] Imaged image M and deblurred image M q The similarity L is comprehensively measured by structural similarity (SSIM) and mean square error (MSE), and its expression is as follows:

[0062]

[0063] Where D is the image M and M q The similarity L is a monotonically increasing function. The larger the value of L, the closer the image M is to the deblurred image M. q The higher the similarity, the better the imaging quality of the current focal length.

[0064] Since here the image M and the deblurred image M q Both are image sets, so the imaging set and the deblurred image set are calculated using the following formula.

[0065]

[0066] S7: Select the focal length interval I with the highest similarity among n pairs of images x , that is, select different imaging intervals to calculate the maximum value of L;

[0067] S8: Determine whether to end the focusing.

[0068] There are two methods to judge the end of focusing. One is to manually set the threshold Y. If I x The similarity L x If it is greater than the threshold value Y, the focus adjustment is exited; if it is greater than the threshold value, the focus adjustment is ended; if it is less than the threshold value Y, I x As a new focusing interval I, steps S2 to S8 are repeated.

[0069] Another way is to record every time I x The similarity L x , that is [L x1 , …, L xn ], if after the n+1th iteration L xn+1 Greater than or equal to L xn The focal length of the nth time is taken as the best focal length and the focusing is exited. xn+1 Less than or equal to L xn Then the I of the n+1th round x As a new focusing interval I, steps S2 to S8 are repeated.

[0070] The above specific examples are used to illustrate the content of the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art of the present invention, according to the idea of ​​the present invention, several simple deductions, modifications or substitutions can be made.

Claims

1. A high-precision and fast focusing method for an optical remote sensing camera based on deep learning, characterized in that: The following steps are involved: Step 1: Deploy the deblurring deep learning model N trained with a large number of samples; Step 2: Determine the focusing range I of the remote sensing camera; Step 3: Divide the focus range I into n range intervals I1, ..., I with a step length of I / n n ; Step 4: The remote sensing camera uses I1,…,I n The midpoint of the interval is the focal length and images are taken n times to obtain n images M1, ..., M n ; Step 5: Transform n images M1, ..., M n They are sent to the deblurring deep learning model N respectively to obtain n corresponding deblurred images M q 1,…,M q n ; Step 6: Calculate the similarity of n pairs of images in step 5 respectively; Step 7: Select the focal length interval I with the highest similarity among n pairs of images x ; Step 8: Determine whether the highest similarity is greater than a threshold. If it is greater than the threshold, the focusing is terminated. If it is less than the threshold, I x As a new focus interval I, continue to repeat steps 2 to 8.

2. The high-precision and fast focusing method for an optical remote sensing camera based on deep learning according to claim 1, characterized in that: The specific implementation process in step 1 includes three processes: data set construction, model N construction, and model training; The dataset consists of A pairs of blurred and clear images. The blurred image is used as the input of the model N, and the clear image is used as the label to measure the deblurring performance of the model. The deblurring model N directly inputs the blurred image M of size H×W and outputs the deblurred image M of size H×W. q The model training uses the loss function Loss to measure the degree of deblurring, and its expression is as follows: A ≥ 1000; Where D represents the grayscale level of image M, M t Indicates clear image labels, SSIM and MSE represent peak-to-peak signal-to-noise ratio and mean square error respectively. The calculation methods of SSIM and MSE are as follows: Where a, b, c are constants, l, c, s represent brightness, contrast and structure respectively, and the calculation formula is as follows: where μ M represents the grayscale mean of the image, σ M represents the standard deviation of the image, Represents the covariance between images, c1, c2, and c3 are constants. The training process is an iterative process, and the model uses loss back propagation to update the weights.

3. The high-precision and rapid focusing method for an optical remote sensing camera based on deep learning according to claim 2, characterized in that: Step 1 During the training process, use the validation set to evaluate the deblurring effect of the model and calculate the deblurred image M q With clear image M t The similarity L of ; similarity L is a monotonically increasing function, and the weight of the model with the highest similarity is retained; the calculation formula of similarity L is as follows; Where D represents the grayscale level of image M.

4. The high-precision and fast focusing method for an optical remote sensing camera based on deep learning according to claim 3, characterized in that: Remote sensing camera image M with grayscale level D and deblurred image M q The similarity L is measured by structural similarity and mean square error, and its expression is as follows: The similarity L is a monotonically increasing function. The larger the value of L, the closer the image M is to the deblurred image M. q The higher the similarity, the better the imaging quality of the current focal length.

5. The high-precision and fast focusing method for an optical remote sensing camera based on deep learning according to claim 2, characterized in that: After step 1 training, use the test set to evaluate the deblurring effect of the model and calculate the deblurred image M q With clear image M t similarity L; if the model deblurring effect meets the focusing requirements, the weight is used to deploy the model, otherwise the model learning rate, step size update strategy and other training parameters are adjusted and retrained.

6. The high-precision and fast focusing method for an optical remote sensing camera based on deep learning according to claim 1, characterized in that: In step 4, the remote sensing camera uses I1,…,I n The midpoint of the interval is the focal length and images are taken n times. The n images do not need to capture the same object, but only need to keep the imaging posture of the remote sensing camera consistent.

7. The high-precision and fast focusing method for an optical remote sensing camera based on deep learning according to claim 1, characterized in that: In step 8, the method for determining the end of focusing is: manually setting the threshold Y, if I x The similarity L x If it is greater than the threshold value Y, the focus adjustment is exited; if it is greater than the threshold value, the focus adjustment is ended; if it is less than the threshold value Y, I x As a new focusing interval I, steps S2 to S8 are repeated.

8. The high-precision and fast focusing method for an optical remote sensing camera based on deep learning according to claim 1, characterized in that: In step 8, the method for judging the end of focusing is to record each I x The similarity L x , that is [L x1 , …, L xn ], if after the n+1th iteration L xn+1 Greater than or equal to L xn The focal length of the nth time is taken as the optimal focal length and the focusing is exited; if L xn+1 Less than or equal to L xn Then the I of the n+1th round x As a new focusing interval I, steps S2 to S8 are repeated.

Images