Autofocus method based on single frame image defocus estimation

Abstract

The present application belongs to the technical field of automatic focusing, and particularly relates to an automatic focusing method based on single-frame image defocus estimation. The method comprises the following steps: S1: performing data preprocessing on a defocus image dataset; S2: obtaining an improved MobileNetV2 network model; S3: constructing a defocus feature vector dataset by using defocus feature vectors of each defocus image; S4: training a gradient boosting regression tree by using the defocus feature vector dataset to obtain a trained gradient boosting regression tree; S5: performing defocus regression prediction by using the improved MobileNetV2 network model and the trained gradient boosting regression tree; and S6: calculating a defocus amount code value according to a defocus amount normalized output value, and adjusting the position of a focusing motor of an optical system according to the defocus amount code value to complete automatic focusing of the optical system. The present application can improve the automatic focusing precision and efficiency of a light measurement imaging device under a dynamic target.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic focusing, and particularly relates to an automatic focusing method based on single-frame image defocus estimation. BACKGROUND

[0002] Clear imaging is the basis of light measurement imaging equipment, and accurate focusing is the key to achieving this goal. In the face of targets with variable positions and different motion states, the optical system must have real-time and rapid automatic focusing ability to maintain the clarity and accuracy of imaging by adjusting the lens position in real time. At present, the popular automatic focusing methods for light measurement imaging equipment include ranging method and digital image-based method. The ranging method uses infrared, laser or ultrasonic ranging devices to measure the distance between the target and the lens (object distance), and then calculates the corresponding image distance according to the Gaussian imaging formula to drive the lens to achieve rapid and accurate focusing, but it is greatly affected by the environment (temperature, air pressure and other environmental factors), and requires additional equipment and higher cost. The digital image-based method only relies on the acquired image information, uses digital image processing technology to analyze the defocus state, controls the focusing mechanism, and effectively searches for the focusing position, mainly including two types of defocus depth method and focus depth method. The defocus depth method estimates the defocus amount by analyzing a small number of defocused images to adjust the image plane position to achieve focusing, but its accuracy is limited by the accuracy of the imaging system parameters and model, and because the model uses an approximate mathematical model, it cannot be accurately determined, resulting in low focusing accuracy. The focus depth method determines the defocus degree and defocus direction of the image using the sharpness evaluation function according to images of different defocus degrees, and uses a focusing search strategy to drive the lens to move in the direction of the evaluation curve peak point until the best focusing position is reached, but the repeated focusing process and the problem of being easily trapped in local extreme values limit its use in light measurement. Therefore, how to improve the automatic focusing accuracy and efficiency of light measurement imaging equipment under dynamic targets is a difficult task. SUMMARY

[0003] Therefore, the present application aims to provide an automatic focusing method based on single-frame image defocus estimation to solve the problem that the prior art cannot improve the automatic focusing accuracy and efficiency of light measurement imaging equipment under dynamic targets, and the present application can improve the automatic focusing accuracy and efficiency of light measurement imaging equipment under dynamic targets.

[0004] To achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0005] An automatic focusing method based on single-frame image defocus estimation, specifically comprising the following steps:

[0006] S1: Construct a defocused image dataset and perform data preprocessing on the defocused image dataset;

[0007] S2: Construct an improved MobileNetV2 network, train the improved MobileNetV2 network using the preprocessed defocus image dataset, and obtain an improved MobileNetV2 network model;

[0008] S3: Construct a defocus feature vector dataset using the defocus feature vectors corresponding to each defocus image in the defocus image dataset output by the improved MobileNetV2 network model;

[0009] S4: Train the gradient boosting regression tree using the defocus feature vector dataset, and obtain a trained gradient boosting regression tree;

[0010] S5: Obtain a to-be-predicted image collected by the optical system, pre-process the to-be-predicted image, and perform defocus regression prediction on the pre-processed to-be-predicted image through the improved MobileNetV2 network model and the trained gradient boosting regression tree to obtain a defocus amount normalized output value corresponding to the to-be-predicted image;

[0011] S6: Calculate a defocus amount code value according to the defocus amount normalized output value, and adjust the position of the focusing motor of the optical system according to the defocus amount code value to complete automatic focusing of the optical system.

[0012] Further, step S1 specifically includes the following steps:

[0013] S11: Align the optical system to the observation scene, adjust the position of the focusing motor according to a preset step size, make the optical system undergo focusing changes from a minimum focusing value to a maximum focusing value at the current observation position, and record the images corresponding to each focusing motor position in sequence to obtain a continuous defocus image stack I;

[0014] S12: Process the continuous defocus image stack using a comprehensive sharpness evaluation method to obtain a focus image;

[0015] S13: Calculate the defocus code values corresponding to each defocus image included in the continuous defocus image stack I based on the focus code value of the focus image, and perform normalization processing on the defocus code values to obtain normalized defocus amount normalized labels in correspondence:

[0016] ;

[0017] Wherein, is the defocus amount corresponding to the i-th defocus image, is the focusing code value of the i-th defocus image, is the focus code value of the focus image;

[0018] ;

[0019] Wherein, is the defocus amount of the i-th defocus image, is a constant, and F max -F min ;

[0020] S14: repeating steps S11-S13 to obtain a normalized defocus image dataset under different observation scenes;

[0021] S15: expanding the defocus image dataset obtained in step S14, and filtering the expanded defocus image dataset using a fast guided filter;

[0022] S16: normalizing each defocus image contained in the filtered defocus image dataset, so that the pixel value of each channel of each defocus image is scaled to the range of [0, 1], completing the data preprocessing of the defocus image dataset.

[0023] Further, in step S12, the calculation formula of the comprehensive sharpness evaluation method is:

[0024] S= + ;

[0025] wherein S is the comprehensive sharpness evaluation value, is the score value of the energy gradient function, is the score value of the Tenengrad function, is the score value of the improved sum of absolute difference of gray value function.

[0026] Further, the calculation formula of the improved sum of absolute difference of gray value function is:

[0027] ;

[0028] wherein the resolution of the defocus image I is , x is the row pixel coordinate position of the defocus image I, y is the column pixel coordinate position of the defocus image I, is the length of the defocus image I, is the width of the defocus image I.

[0029] Further, in step S2, the improved MobileNetV2 network model takes the MobileNetV2 network model as the basic framework, embeds the coordinate attention mechanism module after the depth separable convolution of each bottleneck layer of the MobileNetV2 network model, and embeds the cascaded coordinate attention mechanism module and global average pooling layer after the last convolution layer of the MobileNetV2 network model.

[0030] Further, the coordinate attention mechanism module comprises an X direction average pooling layer, a Y direction average pooling layer, a Concat module, a first 2D convolution layer, a block normalization module, a nonlinear layer, a second 2D convolution layer, a third 2D convolution layer, a first Sigmoid module and a second Sigmoid module, wherein the feature map A input to the coordinate attention mechanism module is input to the X direction average pooling layer and the Y direction average pooling layer for processing, respectively, to obtain the feature map A1 and the feature map A2, the feature map A1 and the feature map A2 are input to the Concat module for processing to obtain the feature map A3, the feature map A3 is input to the first 2D convolution layer for processing to obtain the feature map A4, the feature map A4 is processed by the block normalization module and the nonlinear layer to obtain the feature map A5, the feature map A5 is processed by the split function to obtain the feature map A6, the feature map A6 is input to the second 2D convolution layer and the third 2D convolution layer for processing, respectively, to obtain the feature map A7 and the feature map A8, the feature map A7 is input to the first Sigmoid module to obtain the feature map A9, the feature map A8 is input to the second Sigmoid module to obtain the feature map A10, the feature map A9, the feature map A10 and the feature map A are multiplied to obtain the output feature of the coordinate attention mechanism module.

[0031] Further, in step S3, the out-of-focus feature vector dataset wherein, is the 1st out-of-focus image, is the out-of-focus feature vector corresponding to the 1st out-of-focus image, is the out-of-focus amount normalized label corresponding to the 1st out-of-focus image, is the 2nd out-of-focus image, is the out-of-focus feature vector corresponding to the 2nd out-of-focus image, is the out-of-focus amount normalized label corresponding to the 2nd out-of-focus image, is the nth out-of-focus image, is the out-of-focus feature vector corresponding to the nth out-of-focus image, is the out-of-focus amount normalized label corresponding to the nth out-of-focus image.

[0032] Further, in step S4, the prediction accuracy of the predicted code value output by the trained gradient boosting regression tree is calculated:

[0033] ;

[0034] ;

[0035] wherein, is the accuracy rate, N is the total number of test samples, is the predicted code value error less than the number of samples, is a constant, is a prediction error, is a prediction code value, is an out-of-focus amount normalized label, is a constant.

[0036] Further, the out-of-focus amount code value The calculation formula is:

[0037] .

[0038] Further, in step S5, the specific process of pre-processing the to-be-predicted image is: a fast guided filter is used to perform filtering operation on the to-be-predicted image, and the to-be-predicted image after the filtering operation is normalized, so that the pixel value of each channel of the to-be-predicted image after the filtering operation is scaled to the range of [0, 1].

[0039] Compared with the prior art, the application can achieve the following beneficial effects:

[0040] (1) The automatic focusing method based on single-frame image out-of-focus estimation disclosed by the application regards the focusing process of automatic focusing as a regression problem, designs an improved MobileNetV2 network model to extract features of a significant target area, constructs a strong regressor model of a gradient boosting regression tree (XGBoost), learns the mapping relationship between the features and the out-of-focus amount, so as to realize accurate estimation of the out-of-focus amount of a single-frame image and automatic focusing of an optical system.

[0041] (2) The automatic focusing method based on single-frame image out-of-focus estimation disclosed by the application proposes an improved lightweight MobileNetV2 network model, respectively embeds a coordinate attention mechanism (CA) after a depth separable convolution of a bottleneck layer of the MobileNetV2 network model and after a last convolution layer of the MobileNetV2 network model, enhances the depth feature extraction capability of the significant target area, improves the quality of feature representation, and adds a global average pooling layer at the end of the MobileNetV2 network model, so as to obtain a fixed-length feature vector, which is convenient for subsequent out-of-focus regression prediction.

[0042] (3) The automatic focusing method based on single-frame image defocus estimation of the present application creates a defocus image dataset with defocus amount labels and uses a comprehensive sharpness evaluation method (energy gradient function, Tenengrad function, improved SMD function) to determine the position of the in-focus image; the defocus image dataset is the basis for training the improved MobileNetV2 network model, ensuring that the improved MobileNetV2 network model can learn the relationship between defocus amount and image features. The present application also uses a fast guided filter (FGIF) to preprocess the defocus image, effectively reducing noise while avoiding damage to important structural features such as image edges and details, to improve the feature extraction quality of subsequent defocus images. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application, and their

[0044] Figure 1 Structure schematic diagram of the focusing device according to the embodiment of the present application;

[0045] Figure 2 Flowchart of the automatic focusing method based on single-frame image defocus estimation according to the embodiment of the present application;

[0046] Figure 3 Processing flowchart of the automatic focusing method based on single-frame image defocus estimation according to the embodiment of the present application;

[0047] Figure 4 Network structure schematic diagram of the improved MobileNetV2 network model according to the embodiment of the present application;

[0048] Figure 5 Network structure schematic diagram of the bottleneck layer of the improved MobileNetV2 network model according to the embodiment of the present application.

[0049] BRIEF DESCRIPTION OF DRAWINGS

[0050] 1, optical system; 2, focusing mechanism; 3, focusing controller; 4, image processing software; 41, image acquisition module; 42, image display module; 43, image storage module; 44, image tracking module; 45, defocus prediction module; 46, communication module. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not constitute a limitation on the present application.

[0052] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0053] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only used for description purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0054] In the description of the present application, it should be noted that unless otherwise specified and limited, the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0055] The present application will be described in detail below with reference to the drawings and embodiments.

[0056] As Figure 1As shown, the focusing device includes an optical system 1, a focusing mechanism 2, a focusing controller 3 and image processing software 4, the optical system 1 includes an optical lens and an image detector, the optical system 1 is used for optical signal collection and focusing on the image detector to form an image; the focusing mechanism 2 is used for adjusting the relative position between the lens and the image detector to ensure the image is clear, which belongs to the execution component; the focusing controller 3 mainly controls the work of the focusing mechanism 2, which is the control component; the image processing software 4 mainly includes an image acquisition module 41, an image display module 42, an image storage module 43, an image tracking module 44, an out-of-focus prediction module 45 and a communication module 46; the image acquisition module 41 is used for image acquisition; the image display module 42 is used for displaying the collected image; the image storage module 43 is used for synchronous storage of image data and corresponding focusing feedback code values, and the image storage module 43 is mainly used in the establishment part of the out-of-focus data set; the image tracking module 44 is used for extracting the target tracking wave door to provide the focusing area of interest for the out-of-focus prediction module 45; the out-of-focus amount prediction module is used for predicting the out-of-focus amount of the current image by using the trained model; the communication module 46 encodes the current focusing amount according to the out-of-focus amount and sends the focusing command to the focusing controller 3; the focusing controller 3 sends the driving signal to the focusing mechanism 2 according to the focusing command of the communication module 46, so as to drive the focusing motor to adjust the position and realize the focusing. At the same time, the focusing mechanism 2 feeds back the position of the focusing motor to the image processing software 4 in real time during the focusing process.

[0057] As shown in the Figures 2-3 The application provides an automatic focusing method based on single-frame image out-of-focus estimation, and specifically comprises the following steps: S1, constructing an out-of-focus image data set and performing data preprocessing on the out-of-focus image data set; S2, constructing an improved MobileNetV2 network, training the improved MobileNetV2 network by using the preprocessed out-of-focus image data set, and obtaining an improved MobileNetV2 network model; S3, constructing an out-of-focus feature vector data set by using the out-of-focus feature vectors corresponding to each out-of-focus image in the out-of-focus image data set output by the improved MobileNetV2 network model; S4, training a gradient boosting regression tree by using the out-of-focus feature vector data set, and obtaining a trained gradient boosting regression tree; S5, obtaining a to-be-predicted image collected by an optical system, preprocessing the to-be-predicted image, performing out-of-focus regression prediction on the preprocessed to-be-predicted image by using the improved MobileNetV2 network model and the trained gradient boosting regression tree, and obtaining an out-of-focus amount normalized output value corresponding to the to-be-predicted image; and S6, calculating an out-of-focus amount code value according to the out-of-focus amount normalized output value, adjusting the position of a focusing motor of the optical system according to the out-of-focus amount code value, and completing automatic focusing of the optical system.

[0058] The application converts the automatic focusing problem into a regression problem. First, a defocus image dataset with defocus amount labels is established, and a comprehensive sharpness evaluation method is used to determine the in-focus image. Next, an improved MobileNetV2 network model is proposed, which realizes more concentrated deep feature extraction of the significant target area by introducing a coordinate attention mechanism (CA), and improves the quality of feature representation. Next, the gradient boosting regression tree (XGBoost) is used to train the feature vectors extracted by the improved lightweight MobileNetV2 network model, learn the mapping relationship between the features and the defocus amount, obtain the step and direction required for the focusing process of the optical system 1, reduce the unnecessary reciprocating movement of the optical system 1 in the focusing process, and thus realize accurate estimation of the defocus amount of a single frame image, and automatic focusing of the optical system 1. The application not only ensures the accuracy of image focusing, but also significantly improves the focusing efficiency.

[0059] In some embodiments, step S1 specifically comprises the following steps: S11: aligning the optical system 1 to the observation scene, adjusting the position of the focusing motor according to the preset step size, causing the optical system 1 to undergo focusing changes from the minimum focusing value to the maximum focusing value at the current observation position, and sequentially recording the images corresponding to each focusing motor position to obtain a continuous defocus image stack I; S12: processing the continuous defocus image stack using a comprehensive sharpness evaluation method to obtain an in-focus image; S13: calculating the defocus values corresponding to each defocus image included in the continuous defocus image stack I based on the in-focus value of the in-focus image, and performing normalization processing on each defocus value to correspondingly obtain normalized defocus amount normalization labels; S14: repeating steps S11-S13 to obtain a normalized defocus image dataset under different observation scenes; S15: expanding the defocus image dataset obtained in step S14, and performing filtering operation on the expanded defocus image dataset using a fast guided filter; S16: performing normalization processing on each defocus image included in the defocus image dataset after the filtering operation, so that the pixel values of each channel of each defocus image are scaled to the range of [0, 1], and the data preprocessing of the defocus image dataset is completed.

[0060] It should be noted that in step S11, the position of the focusing motor is usually adjusted according to the preset step size in a manual focusing manner, so that the optical system 1 undergoes focusing changes from the minimum focusing value F min to the maximum focusing value F max at the current observation position. In this process, a continuous defocus image stack from blur-clearness-blur will be obtained.

[0061] Further, in step S13, the in-focus value is F in-focus , and the defocus values corresponding to each defocus image included in the continuous defocus image stack I are i is:

[0062]

[0063] wherein, is the defocus amount corresponding to the i-th defocus image, is the focus code value of the i-th defocus image, is the focus code value of the in-focus image;

[0064] The defocus code values of each defocus image are normalized to make a defocus amount normalized label:

[0065]

[0066] wherein, is the defocus amount normalized label of the i-th defocus image, is a constant, and F max min . In addition, in step S14, in order to ensure the generalization ability of the defocus regression prediction algorithm based on gradient boosting regression tree (XGBoost), the more observation scenes are selected, the better. In step S15, the expansion method of the defocus image dataset is: in order to prevent background interference, the focus region of interest is generally set as the center area of the field of view, and the position of the center area of the field of view is determined according to the specific use. The center region of interest I i of each defocus image I i M×N is randomly cropped to obtain k images I i m×n to increase sample diversity. The fast guided filter can effectively reduce noise while avoiding damage to important structural features such as image edges and details, wherein the filtered image is represented as I , and the guide image uses the input image I i m×n itself.

[0067] In some embodiments, in step S12, the calculation formula of the comprehensive sharpness evaluation method is:

[0068] S= + ;

[0069] wherein S is the comprehensive sharpness evaluation value, is the score value of the energy gradient function, is the score value of the Tenengrad function, is the score value of the improved sum of absolute differences function.

[0070] ​​​It should be noted that, in order to accurately screen the most clear in-focus image and ensure high stability and sensitivity of the automatic focusing method, a comprehensive sharpness evaluation method is used to process the out-of-focus image stack I.

[0071] In some embodiments, the improved calculation formula of the sum of absolute differences of gray scale difference function is:

[0072] ;

[0073] wherein the resolution of the out-of-focus image I is , x is the row pixel coordinate position of the out-of-focus image I, y is the column pixel coordinate position of the out-of-focus image I, is the length of the out-of-focus image I, is the width of the out-of-focus image I.

[0074] It should be noted that the improved sum of absolute differences of gray scale difference function considers the difference between the target pixel point and the adjacent pixel from multiple directions.

[0075] In some embodiments, the improved MobileNetV2 network model takes the MobileNetV2 network model as the basic framework, embeds the coordinate attention mechanism module after the depth separable convolution of each bottleneck layer of the MobileNetV2 network model, and embeds the cascaded coordinate attention mechanism module and the global average pooling layer after the last convolution layer of the MobileNetV2 network model.

[0076] It should be noted that considering that most task scenarios are embedded applications, the limitation of computing resources and storage space is large, therefore the mainstream lightweight MobileNetV2 network model is used as the basic network architecture for feature extraction, and the top fully connected layer is not included. And improve MobileNetV2. Further, the coordinate attention mechanism module (referred to as CA module) is embedded after the depth separable convolution of each bottleneck layer of the MobileNetV2 network model, which compresses the feature map in the spatial dimension to capture the position information, and then generates attention weights through a series of convolution layers. The generated attention weights are used to reweight the original feature map to emphasize the features of important positions and suppress the features of unimportant positions, thereby improving the quality of feature representation. Further, the CA module is embedded again after the last convolution layer of the MobileNetV2 network model, which can further refine the feature map, so that the improved MobileNetV2 network model can pay more attention to key features before making a final decision. Help the improved MobileNetV2 network model to better adapt to different degrees of blur, facilitate subsequent regression tasks. In addition, a global average pooling layer is added after the output of the last convolution layer of the improved MobileNetV2 network model to reduce the spatial dimension of the features and obtain a fixed-length out-of-focus feature vector.

[0077] As shown in Figure 4 , the improved MobileNetV2 network model (abbreviated as MobileNetV2-CA in Figure 3 ) includes a 2d convolution layer, seven bottleneck layers embedded with CA modules, a 2d convolution layer, a CA module, and a global average pooling layer connected in turn, the features input to the improved MobileNetV2 network model are output as feature F through the 2d convolution layer and the seven new bottleneck layers embedded with CA modules, and the output feature is obtained after the feature F is processed by the 2d convolution layer, the CA module and the global average pooling layer.

[0078] As shown in Figure 5 , the bottleneck layer of the improved MobileNetV2 network model is a 1x1 convolution layer, a 3x3 depth convolution layer, a coordinate attention mechanism module (CA module), and a 1x1 convolution layer connected in turn, and the input feature is processed by the 1x1 convolution layer, the 3x3 depth convolution layer, the coordinate attention mechanism module and the 1x1 convolution layer to obtain the intermediate feature. Add the intermediate feature and the input feature to obtain the output feature of the bottleneck layer of the improved MobileNetV2 network model.

[0079] In some examples, the coordinate attention mechanism module comprises an X direction average pooling layer, a Y direction average pooling layer, a Concat module, a first 2D convolution layer, a block normalization module, a nonlinear layer, a second 2D convolution layer, a third 2D convolution layer, a first Sigmoid module and a second Sigmoid module, wherein the feature map A input to the coordinate attention mechanism module is input to the X direction average pooling layer and the Y direction average pooling layer for processing, respectively, to obtain the feature map A1 and the feature map A2, the feature map A1 and the feature map A2 are input to the Concat module for processing to obtain the feature map A3, the feature map A3 is input to the first 2D convolution layer for processing to obtain the feature map A4, the feature map A4 is processed by the block normalization module and the nonlinear layer to obtain the feature map A5, the feature map A5 is processed by the split function to obtain the feature map A6, the feature map A6 is input to the second 2D convolution layer and the third 2D convolution layer for processing, respectively, to obtain the feature map A7 and the feature map A8, the feature map A7 is input to the first Sigmoid module to obtain the feature map A9, the feature map A8 is input to the second Sigmoid module to obtain the feature map A10, the feature map A9, the feature map A10 and the feature map A are multiplied to obtain the output feature of the coordinate attention mechanism module.

[0080] In some examples, in step S3, the out-of-focus feature vector dataset wherein, is the 1st out-of-focus image, is the out-of-focus feature vector corresponding to the 1st out-of-focus image, is the out-of-focus amount normalized label corresponding to the 1st out-of-focus image, is the 2nd out-of-focus image, is the out-of-focus feature vector corresponding to the 2nd out-of-focus image, is the out-of-focus amount normalized label corresponding to the 2nd out-of-focus image, is the nth out-of-focus image, is the out-of-focus feature vector corresponding to the nth out-of-focus image, is the out-of-focus amount normalized label corresponding to the nth out-of-focus image.

[0081] It should be noted that in step S4, gradient boosting regression tree (XGBoost) is an ensemble learning method that constructs a powerful prediction model by integrating multiple weak classifiers, and by continuously adding tree models, each time adding a new tree to fit the residual of the last prediction, and finally adding the prediction results of all trees to obtain the final prediction code value.

[0082] The training process of the gradient boosting regression tree is:

[0083] The defocus feature vectors corresponding to each defocus image contained in the defocus image dataset output by the improved MobileNetV2 network model are proportionally divided into a training set, a validation set and a test set, and the division proportions are 7:2:1 respectively.

[0084] The number of decision trees is set to 200, the learning rate is set to 0.1, the maximum depth of the decision tree is set to 16, and the loss function is selected to be the minimum square error loss.

[0085] The initial model of the decision tree is set :

[0086] ;

[0087] wherein, is the average value of the normalized defocus amount label in the training set, A is the total number of samples in the training set, is the i-th normalized defocus amount label;

[0088] Calculate the residual of each defocus feature vector under the current model:

[0089] ;

[0090] wherein, is the residual, is the current regression tree model, is the input defocus feature vector; for the square loss function, the residual is the difference between the target value and the current predicted code value.

[0091] Fit the regression tree: use the residual as the target variable, train a regression tree T to fit the data, and the regression tree divides the input space into multiple regions.

[0092] Calculate the leaf node value: for each leaf node of the regression tree T, find the optimal output value by minimizing the loss function:

[0093] Calculate each leaf node of the decision tree, minimize the square loss function, and obtain:

[0094] ;

[0095] wherein, is the optimal output value of each leaf node, is the set of defocus feature vectors in the j-th leaf node, L() is the loss function, is a constant.

[0096] Update the model: add the newly trained regression tree to the model and update the predicted code value of the model:

[0097] ;

[0098] where, is the learning rate, is the predicted code value of the z-th tree for x, is the parameter of the regression tree T.

[0099] Repeat the above steps until a predetermined number of iterations Z is reached, or when adding a new tree no longer significantly reduces the loss function, resulting in a final strong regression tree model:

[0100]

[0101] This process generates an additive model consisting of multiple trees, each trained based on the residual of the previous tree, gradually building a strong regression model.

[0102] The performance of the gradient boosting regression tree is evaluated using the validation set, with the performance metrics using mean squared error (MSE) and decision coefficient (r2_score).

[0103] After the training of the improved MobileNetV2 network model and the gradient boosting regression tree is completed, the defocus amount prediction evaluation is carried out: the accuracy of defocus amount prediction is evaluated using the test set. Considering that the focusing system of the light measuring device adopts a high-precision motor drive, the lens is focused in a large stroke range, and the focusing value has high precision. If the error of defocus amount prediction is within the range of image clarity that cannot be distinguished by the human eye, it is generally acceptable. Such error has little effect on actual observation tasks. In this case, even if there is a certain prediction error, the image is still considered to be clear enough to meet most observation needs. Therefore, the accuracy of defocus amount prediction is as follows:

[0104] ;

[0105] where, is the accuracy, N is the total number of test samples, is the number of samples with a prediction code value error less than is a constant, is the prediction error, is the predicted code value, is the defocus amount normalized label, is a constant, which is determined according to the situation. represents the prediction error, and the expression is:

[0106] ;

[0107] where, ​a predicted code value of the defocus amount, a normalized label of the defocus amount, is a constant. It is tested that more than 95% can be achieved, and the performance of the present application is considered to be good. After the training is completed, the model (improved MobileNetV2 network model and trained gradient boosting regression tree) is saved for subsequent use or deployment.

[0108] The trained model is deployed into the focusing control software. If the tracking wave gate of the image tracking module 44 is received, the tracking target area is set as the focusing region of interest, and if the tracking wave gate of the image tracking module 44 is not received, the image center area is set as the focusing region of interest. The focusing region of interest image of the current frame image collected is input into the defocus prediction module 45 for defocus amount prediction. The prediction steps are as shown in Figure 4 First, the region of interest image is preprocessed, and then the improved MobileNetV2 network model is used for defocus feature extraction to obtain a feature vector, which is then output to the XGBoost regression model to obtain the defocus amount normalized output value (with positive and negative values in direction) of the current image position, and converted into a defocus amount code value by the formula When Coder <0, the focusing motor is controlled to adjust in the F max direction, and when >0, the focusing motor is controlled to adjust in the F Coder direction, and the code value of the adjustment is the absolute value of min , that is, the automatic focusing of the system can be completed.

[0109] It should be understood that various forms of the flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0110] The above specific embodiments do not constitute a limitation on the scope of protection of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An auto-focusing method based on single-frame image defocus estimation, characterized in that: Specifically comprising the following steps: S1: Constructing an out-of-focus image dataset and performing data preprocessing on the out-of-focus image dataset; Step S1 specifically comprises the following steps: S11: Aligning the optical system to the observation scene, adjusting the position of the focusing motor according to a preset step size, causing the optical system to undergo focusing changes from the minimum focusing value to the maximum focusing value at the current observation position, and sequentially recording the images corresponding to each focusing motor position to obtain a continuous out-of-focus image stack I; S12: Processing the continuous out-of-focus image stack using a comprehensive sharpness evaluation method to obtain a focus image; In step S12, the calculation formula of the comprehensive sharpness evaluation method is: S= + ； wherein S is a comprehensive sharpness evaluation value, is a score value of an energy gradient function, is a score value of a Tenengrad function, is a score value of an improved sum of absolute difference of gray value function; The calculation formula of the improved sum of absolute differences function is: ； wherein the resolution of the defocus image I is , x is a row pixel coordinate position of the defocus image I, y is a column pixel coordinate position of the defocus image I, is a length of the defocus image I, is a width of the defocus image I; S13: Calculating the defocus code value corresponding to each out-of-focus image contained in the continuous out-of-focus image stack I based on the focus code value of the focus image, and performing normalization processing on each defocus code value to obtain a normalized defocus amount normalization label: ； wherein, is the defocus amount corresponding to the i-th defocused image, is the focus code value of the i-th defocused image, is the focus code value of the in-focus image; ； wherein, is a defocus amount normalized label for the i-th defocused image, is a constant, taken F max -F min ; S14: Repeating steps S11-S13 to obtain a normalized out-of-focus image dataset under different observation scenes; S15: Extending the out-of-focus image dataset obtained in step S14, and filtering the extended out-of-focus image dataset using a fast guided filter; S16: Normalizing each out-of-focus image contained in the filtered out-of-focus image dataset so that the pixel values of each channel of each out-of-focus image are scaled to the range [0, 1], completing the data preprocessing of the out-of-focus image dataset; S2: Constructing an improved MobileNetV2 network, training the improved MobileNetV2 network using the preprocessed out-of-focus image dataset, and obtaining an improved MobileNetV2 network model; In step S2, the improved MobileNetV2 network model takes the MobileNetV2 network model as the basic framework, embeds a coordinate attention mechanism module after the depth separable convolution of each bottleneck layer of the MobileNetV2 network model, and embeds a cascaded coordinate attention mechanism module and a global average pooling layer after the last convolution layer of the MobileNetV2 network model; S3: Constructing an out-of-focus feature vector dataset using the out-of-focus feature vectors corresponding to each out-of-focus image contained in the out-of-focus image dataset output by the improved MobileNetV2 network model; S4: Training a gradient boosting regression tree using the out-of-focus feature vector dataset to obtain a trained gradient boosting regression tree; S5: Obtaining a to-be-predicted image collected by the optical system, preprocessing the to-be-predicted image, and performing out-of-focus regression prediction on the preprocessed to-be-predicted image using the improved MobileNetV2 network model and the trained gradient boosting regression tree to obtain a defocus amount normalization output value corresponding to the to-be-predicted image; S6: Calculating a defocus amount code value according to the defocus amount normalization output value, and adjusting the position of the focusing motor of the optical system according to the defocus amount code value to complete the automatic focusing of the optical system.

2. The autofocus method based on single-frame image defocus estimation according to claim 1, characterized in that: The coordinate attention mechanism module comprises an X-direction average pooling layer, a Y-direction average pooling layer, a Concat module, a first 2D convolution layer, a block normalization module, a nonlinear layer, a second 2D convolution layer, a third 2D convolution layer, a first Sigmoid module and a second Sigmoid module, wherein the feature map A input to the coordinate attention mechanism module is input to the X-direction average pooling layer and the Y-direction average pooling layer for processing, and the feature map A1 and the feature map A2 are obtained correspondingly, the feature map A1 and the feature map A2 are input to the Concat module for processing, and the feature map A3 is obtained, the feature map A3 is input to the first 2D convolution layer for processing, and the feature map A4 is obtained, the feature map A4 is processed through the block normalization module and the nonlinear layer, and the feature map A5 is obtained, the feature map A5 is processed through the split function, and the feature map A6 is obtained, the feature map A6 is input to the second 2D convolution layer and the third 2D convolution layer for processing, and the feature map A7 and the feature map A8 are obtained correspondingly, the feature map A7 is input to the first Sigmoid module, and the feature map A9 is obtained, the feature map A8 is input to the second Sigmoid module, and the feature map A10 is obtained, the feature map A9, the feature map A10 and the feature map A are multiplied, and the output feature of the coordinate attention mechanism module is obtained.

3. The single-frame image defocus estimation based auto-focusing method according to claim 1, wherein: In step S3, the out-of-focus feature vector dataset wherein, is the 1st out-of-focus image, is the out-of-focus feature vector corresponding to the 1st out-of-focus image, is the out-of-focus amount normalized label corresponding to the 1st out-of-focus image, is the 2nd out-of-focus image, is the out-of-focus feature vector corresponding to the 2nd out-of-focus image, is the out-of-focus amount normalized label corresponding to the 2nd out-of-focus image, is the n-th out-of-focus image, is the out-of-focus feature vector corresponding to the n-th out-of-focus image, y n is the out-of-focus amount normalized label corresponding to the n-th out-of-focus image.

4. The single-frame image defocus estimation based auto-focusing method according to claim 1, wherein: In step S4, the prediction accuracy of the predicted code value output by the trained gradient boosting regression tree is calculated: ； ； wherein, is the accuracy rate, N is the total number of test samples, is the number of samples with a predicted code value error less than , is a constant, is the prediction error, is the predicted code value, is the defocus amount normalized label, is a constant.

5. The single-frame image defocus estimation based auto-focusing method according to claim 4, characterized in that: Defocus amount code value The calculation formula is: 。 6. The single-frame image defocus estimation based auto-focusing method according to claim 1, wherein: In step S5, the specific process of pre-processing the to-be-predicted image is: a fast guided filter is used to perform filtering operation on the to-be-predicted image, and the to-be-predicted image after the filtering operation is normalized, so that the pixel value of each channel of the to-be-predicted image after the filtering operation is scaled to the range of [0, 1].

Images