A split-image autofocus method based on deep learning

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a split-image autofocus method based on deep learning, comprising: collecting an original image at the current position of a microscope; extracting an image sub-block where a split-image pattern is located in the original image; repairing the image sub-block using a trained deep learning model to obtain a repaired image sub-block; calculating the relative pixel distance between the left and right split-image patterns in the repaired image sub-block using a similarity algorithm; predicting the defocus distance of the original image from the relative pixel distance based on a pre-calibrated relationship model; and moving the objective lens to a target position based on the predicted defocus distance to complete the autofocus of the microscope. The present invention achieves focusing from a single image, greatly improving the speed of the microscope's autofocus, and using a deep learning model to repair the image during prediction can better combat interference caused by complex foregrounds in the image, effectively improving the accuracy and robustness of autofocus.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly, to a split-image autofocus method based on deep learning. Background Art

[0002] Autofocus technology is crucial in the microscope imaging process. It not only reduces the subjective errors caused by manual focus but also significantly improves focusing speed. Autofocus technology can be divided into active and passive focus technologies. Active focus technology requires the installation of an additional distance sensor on the microscope to directly measure the distance between the objective lens and the sample. Passive focus technology uses digital images to predict the amount of defocus.

[0003] Passive autofocus methods based on digital image processing are currently commonly used in microscopes. This method includes an image clarity evaluation algorithm and an iterative search algorithm. The image clarity algorithm evaluates the clarity of the image captured at the current position to infer its degree of defocus. Combined with the iterative search algorithm, the algorithm maximizes image contrast with the minimum number of movements to obtain the ideal focal plane.

[0004] In industrial production, it's often necessary to inspect the surfaces of processed products because they often contain defects, noise, dust, and other impurities. When digital image processing-based autofocus methods use a clarity evaluation function to evaluate images at different focal planes, impurities, as foreground elements in the image, interfere with the curve of the evaluation function, causing it to no longer meet the requirements of unimodality and unbiasedness, often resulting in multiple peaks and offsets. Consequently, in many scenarios, objects like noise and dust are mistakenly identified as the in-focus subject, seriously impacting autofocus accuracy. Furthermore, in different scenes, lighting conditions, and objective lens magnifications, the images of various complex foreground elements in the camera are uncertain, creating difficulties for subsequent image processing.

[0005] The above-mentioned autofocus method based on digital image processing requires the acquisition of multiple images, which makes the focusing process time-consuming, and is sensitive to complex foregrounds formed by defects, noise, dust and other impurities, resulting in poor accuracy and robustness. Therefore, it is necessary to develop a high-performance autofocus algorithm based on digital image processing to make it faster, more accurate and more robust. Summary of the Invention

[0006] In response to the technical problems existing in the above-mentioned microscope autofocus, the present invention provides a split-image autofocus method based on deep learning to solve the technical problems of existing microscope autofocus such as slow speed, low accuracy and poor robustness.

[0007] As a first aspect of the present invention, a split-image autofocus method based on deep learning is provided, comprising the following steps:

[0008] Step S1: collecting the original image at the current position of the microscope;

[0009] Step S2: extracting the image sub-block where the split-image pattern is located in the original image;

[0010] Step S3: using the trained deep learning model to repair the image sub-block to obtain a repaired image sub-block;

[0011] Step S4: Calculating the relative pixel distance between the left and right split-image patterns in the restored image sub-block using a similarity algorithm;

[0012] Step S5: predicting the defocus distance of the original image from the relative pixel distance according to the pre-calibrated relationship model;

[0013] Step S6: According to the predicted defocus distance, the objective lens is moved to the target position to complete the automatic focusing of the microscope.

[0014] Furthermore, extracting the image sub-block where the split-image pattern is located in the original image includes:

[0015] Stretching the contrast of the original image to between 0 and 255;

[0016] Select a specific threshold T to segment the foreground and background of the original image;

[0017] The center position of the foreground in the original image is located, and an image sub-block with a size of 1536×1536 is extracted based on the center position of the foreground.

[0018] Furthermore, before using the trained deep learning model to repair the image sub-block, the method includes:

[0019] Build a deep learning model based on generative adversarial networks;

[0020] Producing paired data samples for training deep learning models;

[0021] The deep learning model based on the generative adversarial network is trained according to the paired data samples to generate the trained deep learning model.

[0022] Furthermore, the deep learning model based on the generative adversarial network includes a generator and a discriminator, the generator is based on a deep separable convolutional network, and the discriminator is PatchGAN;

[0023] The generator first extracts image features by downsampling the received image three times, and then generates a repaired image by upsampling it three times;

[0024] The discriminator inputs an image, which is downsampled three times and then processed by a fully connected layer, and the evaluation value is output by a Sigmoid function.

[0025] Furthermore, the step of preparing paired data samples for training a deep learning model includes:

[0026] The step length and the distance are fixed to collect images of each focal plane under different foregrounds, and the image sub-blocks where the split-image pattern is located in all images are extracted to obtain multiple sets of focal stacks;

[0027] Determining a reference image sub-block, i.e., an ideal focal plane, for each focus stack according to a judgment criterion, and marking actual defocus distances for other image sub-blocks in each focus stack according to the reference image sub-block;

[0028] The image sub-blocks marked with the same defocus distance in all focus stacks are grouped into one category;

[0029] In each type of image sub-block, all clean foreground images are averaged to fit a new image as the label image of each type of image sub-block;

[0030] Each complex foreground image and the label image corresponding to its class are combined into a pair of data samples to produce all paired data samples.

[0031] Furthermore, the training of the deep learning model based on the generative adversarial network according to the paired data samples to generate the trained deep learning model includes training of a generator and training of a discriminator:

[0032] The training of the generator includes:

[0033] Inputting the complex background image in the paired data sample into the generator, outputting the restored image, and inputting the restored image into the discriminator, outputting an evaluation value;

[0034] A total loss function is obtained according to the root mean square loss of the restored image and the label image and the binary cross entropy loss of the output evaluation value, and the two are weighted in different ratios;

[0035] Update the generator's network parameters according to the total loss function, and repeat the above process until the generator is trained;

[0036] The training of the discriminator includes:

[0037] The label image and the complex background image in the paired data sample are input into the discriminator as true and false images respectively, and the network parameters of the discriminator are updated according to the binary cross entropy loss of the output evaluation value. The above process is repeated until the discriminator is trained.

[0038] Furthermore, determining the reference image sub-blocks of each group of focal stacks according to a judgment criterion includes:

[0039] Using a similarity algorithm to estimate the relative pixel distances between the left and right split-image patterns of all image sub-blocks in each focus stack, selecting the image sub-block with the smallest relative pixel distance as the initial reference image sub-block, and labeling the true defocus distances of the other image sub-blocks in each focus stack according to the initial reference image sub-block;

[0040] According to the pre-calibrated relationship model, the defocus distance of each image sub-block is predicted by the relative pixel distance. The mean square error between the actual defocus distance and the predicted defocus distance corresponding to the 10 image sub-blocks to the left and right of the initial reference image sub-block is calculated.

[0041] Select the left and right image sub-blocks of the initial reference image sub-block as new reference image sub-blocks, and calculate their respective mean square errors according to the above steps;

[0042] Compare the mean square errors of 5 different reference image sub-blocks, and select the image sub-block with the smallest mean square error as the final reference image sub-block.

[0043] Furthermore, the calculating the relative pixel distance between the left and right split-image patterns in the restored image sub-block by using a similarity algorithm includes:

[0044] Segmenting all pixels occupied by the right split-image pattern in the restored image sub-block, calculating the centroid position of the right split-image pattern, and vertically flipping the right split-image pattern;

[0045] The centroid position of the left split-image pattern is located using a similarity algorithm based on the flipped right split-image pattern;

[0046] The relative pixel distance between the left and right split-image patterns is obtained according to the centroid positions of the two.

[0047] Furthermore, predicting the defocus distance of the original image from the relative pixel distance based on the pre-calibrated relationship model requires calibrating the relationship model between the relative pixel distance and the actual defocus distance in advance, including:

[0048] Build a relational model based on gradient boosted decision trees;

[0049] Create a dataset for calibrating the relational model;

[0050] The gradient boosting decision tree-based relationship model is calibrated according to the data set to generate a calibrated relationship model.

[0051] Furthermore, the step of preparing a data set for learning a relational model includes:

[0052] The step length and the stroke are fixed to collect images of each focal plane under the clean foreground, and the image sub-blocks where the split-image pattern is located in all images are extracted to obtain multiple sets of focus stacks;

[0053] The similarity algorithm is used to estimate the relative pixel distances between the left and right split-image patterns in all image sub-blocks;

[0054] Determine the reference image sub-block of each focus stack, i.e., the ideal focal plane, according to the judgment criterion, and mark the actual defocus distances of other image sub-blocks in each focus stack according to the reference image sub-block;

[0055] The relative pixel distance and true defocus distance corresponding to each image sub-block are used as a pair of data samples to produce a dataset of all image sub-blocks.

[0056] The present invention provides a split-image autofocus method based on deep learning, which has the following beneficial effects:

[0057] (1) The present invention can quickly complete focusing by estimating the defocus distance from a single image, greatly improving the speed of microscope autofocus and getting rid of the disadvantage of the traditional method of slow focusing caused by the need to collect multiple images;

[0058] (2) The present invention uses a deep learning model to repair the image and then processes the repaired image, which can better resist the interference caused by complex foreground and effectively improve the accuracy and robustness of autofocus;

[0059] (3) The present invention uses active laser illumination to generate split-image images rather than ordinary natural images. Therefore, the analysis of fixed features in the image has higher accuracy and robustness and is applicable to more scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0061] Figure 1 This is a flowchart of the deep learning-based split-image autofocus method provided by the present invention.

[0062] Figure 2 This is a network structure diagram of the generator in the deep learning model provided by the present invention.

[0063] Figure 3 This is a network structure diagram of the discriminator in the deep learning model provided by the present invention.

[0064] Figure 4 This is a training flowchart for the deep learning model provided by the present invention.

[0065] Figure 5 This is the effect diagram of repairing part of the image using the deep learning model provided by the present invention.

[0066] Figure 6 This is a flow chart of the specific implementation effect provided by the present invention. DETAILED DESCRIPTION

[0067] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a deep learning-based split-image autofocus method proposed by the present invention. Obviously, the described embodiments are only a portion of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments derived by persons of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.

[0068] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0069] In this embodiment, a split-image autofocus method based on deep learning is provided. Figure 1 Please also refer to Figure 6 , a split-image autofocus method based on deep learning, comprising the following steps:

[0070] Step S1: collecting the original image at the current position of the microscope;

[0071] Step S2: extracting the image sub-block where the split-image pattern is located in the original image;

[0072] Preferably, extracting the image sub-block where the split-image pattern is located in the original image includes:

[0073] Stretching the contrast of the original image to between 0 and 255;

[0074] Select a specific threshold T to segment the foreground and background of the original image;

[0075] The center position of the foreground in the original image is located, and an image sub-block with a size of 1536×1536 is extracted based on the center position of the foreground.

[0076] Step S3: Use the trained deep learning model to repair the image sub-block to obtain the repaired image sub-block; compare the image sub-block before and after repair with reference to Figure 5 .

[0077] Preferably, before using the trained deep learning model to repair the image sub-block, the method includes:

[0078] Build a deep learning model based on generative adversarial networks;

[0079] Producing paired data samples for training deep learning models;

[0080] The deep learning model based on the generative adversarial network is trained according to the paired data samples to generate the trained deep learning model.

[0081] Specifically, the deep learning model based on the generative adversarial network includes a generator and a discriminator, the generator is based on a deep separable convolutional network, and the discriminator is PatchGAN;

[0082] like Figure 2 As shown in the figure, the overall structure of the generator includes: first downsampling the input image three times to extract image features, then performing high-dimensional feature extraction by 8 double-layer inverse residual structures, and finally generating a repaired image by upsampling three times and combining the low-dimensional features extracted during downsampling.

[0083] The generator's network architecture consists of two basic blocks. The first is an inverted residual structure, which first uses 1×1 convolution for dimensionality increase, then performs feature extraction through 3×3 depthwise separable convolution, and finally performs dimensionality reduction through 1×1 convolution. The second block directly performs feature extraction through 3×3 depthwise separable convolution, and then adjusts the number of output channels through 1×1 convolution.

[0084] like Figure 3As shown in the figure, the discriminator uses a standard convolutional neural network consisting of five convolutional layers. The first four convolutional layers are each followed by a corresponding instance normalization layer and a LeakyReLU activation function. The last convolutional layer is connected to a pooling layer and then uses a sigmoid activation function to predict the probability value. The first three convolutional layers use downsampling to extract higher-dimensional features of the image, and the last two convolutional layers integrate all the extracted features.

[0085] The generator first extracts image features by downsampling the received image three times, and then generates a repaired image by upsampling it three times;

[0086] The discriminator inputs an image, which is downsampled three times and then processed by a fully connected layer, and the evaluation value is output by a Sigmoid function.

[0087] Specifically, the step of preparing paired data samples for training a deep learning model includes:

[0088] Fixed step size and stroke length were used to capture images of each focal plane under different foreground conditions. Image sub-blocks containing split-image patterns were extracted from all images. The size of each image sub-block was 1536×1536, resulting in 60 focus stacks, including 50 focus stacks with complex foregrounds and 10 focus stacks with clean foregrounds.

[0089] Determining a reference image sub-block, i.e., an ideal focal plane, for each focus stack according to a judgment criterion, and marking actual defocus distances for other image sub-blocks in each focus stack according to the reference image sub-block;

[0090] The image sub-blocks marked with the same defocus distance in all focus stacks are grouped into one category;

[0091] In each type of image sub-block, all clean foreground images are averaged to fit a new image as the label image of each type of image sub-block;

[0092] Each complex foreground image and the label image corresponding to its class form a pair of data samples to produce all paired data samples. All paired data sets are randomly divided into training set, validation set and test set in the ratio of 8:1:1.

[0093] Furthermore, determining the reference image sub-blocks of each group of focal stacks according to a judgment criterion includes:

[0094] Using a similarity algorithm to estimate the relative pixel distances between the left and right split-image patterns of all image sub-blocks in each focus stack, selecting the image sub-block with the smallest relative pixel distance as the initial reference image sub-block, and labeling the true defocus distances of the other image sub-blocks in each focus stack according to the initial reference image sub-block;

[0095] According to the pre-calibrated relationship model, the defocus distance of each image sub-block is predicted by the relative pixel distance. The mean square error between the actual defocus distance and the predicted defocus distance corresponding to the 10 image sub-blocks to the left and right of the initial reference image sub-block is calculated.

[0096] Select the left and right image sub-blocks of the initial reference image sub-block as new reference image sub-blocks, and calculate their respective mean square errors according to the above steps;

[0097] Compare the mean square errors of the five different reference image sub-blocks and select the image sub-block with the smallest mean square error as the final reference image sub-block. The expression is:

[0098]

[0099] Among them, V k is the mean square error between the actual defocus distance and the predicted defocus distance calculated under different reference image sub-blocks, I k are different reference image sub-blocks.

[0100] Specifically, the deep learning model based on the adversarial generative network is trained according to the paired data samples in a supervised manner to generate the trained deep learning model, such as Figure 4 As shown, it includes the training of the generator and the training of the discriminator:

[0101] The training of the generator includes:

[0102] Inputting the complex background image in the paired data sample into the generator, outputting the restored image, and inputting the restored image into the discriminator, outputting an evaluation value;

[0103] According to the root mean square loss of the restored image and the label image and the binary cross entropy loss of the output evaluation value, the total loss function of the generator is obtained according to different weight ratios of the two.

[0104] Update the generator's network parameters according to the total loss function, and repeat the above process until the generator is trained;

[0105] The training of the discriminator includes:

[0106] The label image in the paired data sample and the image generated by the generator are respectively input into the discriminator as true and false images, and the network parameters of the discriminator are updated according to the binary cross entropy loss of the discriminator output evaluation value. The above process is repeated until the discriminator is trained.

[0107] The loss function L of the generator G and the discriminator's loss function L DRespectively expressed as:

[0108]

[0109]

[0110] Where x is the defocused input image, y is the focused label image, N is the batch size, and the weight coefficients of λ and μ are 1 and 10 respectively. -3 , G(x) is the output image of the generator, and D(*) is the evaluation value output by the discriminator.

[0111] Step S4: Calculating the relative pixel distance between the left and right split-image patterns in the restored image sub-block using a similarity algorithm;

[0112] Preferably, the calculating the relative pixel distance between the left and right split-image patterns in the restored image sub-block by using a similarity algorithm comprises:

[0113] Segment all pixels occupied by the right split-image pattern in the repaired image sub-block and calculate the centroid position of the right split-image pattern, denoted as P r , while flipping the right split-image pattern vertically;

[0114] According to the flipped right split image pattern, the normalized cross-correlation similarity algorithm is used to locate the centroid position of the left split image pattern, which is recorded as P l The normalized cross-correlation similarity algorithm (NCC) can be expressed as:

[0115]

[0116]

[0117]

[0118] Where M and N are the pixel coordinate value sets selected from the right split-image, and T is the total number of pixels in the set. L(x,y) and are the pixel value and pixel average of the left split image respectively. R(w,h) and are the pixel value and pixel average of the right split-image image, respectively.

[0119] According to the centroid positions of the left and right split-image patterns, the relative pixel distance between the two is:

[0120] Δpixel=(P l -P r ) v .

[0121] Among them, P l and P r are the centroid coordinates of the left and right split-image patterns, respectively, *v is the coordinate along the v-axis.

[0122] Step S5: predicting the defocus distance Δz of the original image from the relative pixel distance Δpixel according to the pre-calibrated relationship model;

[0123] Preferably, predicting the defocus distance of the original image from the relative pixel distance based on the pre-calibrated relationship model requires calibrating the relationship model between the relative pixel distance and the actual defocus distance in advance, including:

[0124] Build a relational model based on gradient boosted decision trees;

[0125] Create a dataset for calibrating the relational model;

[0126] The gradient boosting decision tree-based relationship model is calibrated according to the data set to generate a calibrated relationship model.

[0127] Specifically, preparing a data set for learning a relational model includes:

[0128] The step size and stroke length are fixed to collect images of each focal plane in a clean foreground. The image sub-blocks where the split-image pattern is located are extracted from all images. The size of each image sub-block is 1536×1536, and 8 sets of focus stacks are obtained.

[0129] The normalized cross-correlation similarity algorithm is used to estimate the relative pixel distances of the left and right split-image patterns in all image sub-blocks.

[0130] Determine the reference image sub-block of each focus stack, i.e., the ideal focal plane, according to the judgment criterion, and mark the actual defocus distances of other image sub-blocks in each focus stack according to the reference image sub-block;

[0131] The relative pixel distance and true defocus distance corresponding to each image sub-block are used as a pair of data samples to produce a dataset of all image sub-blocks.

[0132] The relational model based on the gradient boosting decision tree is calibrated according to the data set. The weight update strategy of the relational model in each round of iterative training can be expressed as:

[0133] F m (x) = F m-1 (x)+v·γ m ·h m (x)

[0134] Among them, F m-1 (x) is the previous model, v is the learning rate, h m (x) is a weak learner, γ m Expressed as:

[0135]

[0136] Among them, (x i ,y i ) is the training data set, γ is the weight, and L is the square loss function expressed as:

[0137]

[0138] The minimum loss is determined by the negative gradient of the loss function L, and the residual r mi Expressed as:

[0139]

[0140] The new model updates its weights along the gradient direction to reduce the residual error of the previous model. After 1000 iterations of training, the new model obtained is the calibrated relationship model.

[0141] Step S6: According to the predicted defocus distance, the objective lens is moved to the target position to complete the automatic focusing of the microscope.

[0142] Specifically, the similarity algorithm is normalized cross-correlation.

[0143] Experimental analysis:

[0144] The experimental environment configuration for this paper was: Windows 10 system, an Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz, 16.0GB of RAM, an NVIDIA 1650 graphics card, Python 3.7, and the TensorFlow framework. In the experiment, the batch size was set to 4, and 30 epochs were run. The learning rate was halved every five epochs. The entire deep learning model was optimized using the Adam optimizer.

[0145] In order to demonstrate the autofocus performance of the present invention, four of the most advanced methods are selected for comparison: the traditional mainstream method Baseline, the ordered regression model I1 model and D1 model, and the regression model Regression model. The comparison results are shown in Table 1. As can be seen from Table 1, the autofocus method of the present invention has greatly improved in focusing accuracy and robustness compared with the other four most advanced methods, especially when the defocus amount is small. Within the range of ±200μm, the mean square error (MAE) of the autofocus method of the present invention is 0.945, which is 1 times higher than the 1.978 of the other four best methods. And within the range of ±480μm with a larger defocus amount, the autofocus method of the present invention also improves the mean square error from the original 2.114 to 1.505. The root mean square error (RMSE) reflects that the autofocus method of the present invention has higher stability for defocus distances predicted by different complex foregrounds than the other four methods. Within the range of ±200μm, the root mean square error is improved from the original 2.3 to 1.185. Therefore, the split-image autofocus method based on deep learning of the present invention is a high-performance autofocus method, which not only has higher focusing accuracy but also has stronger robustness to different complex foregrounds.

[0146] Table 1 Comparison results between the present invention and four advanced methods

[0147]

[0148] In summary, the present invention can quickly complete focusing by estimating the defocus distance from a single image, greatly improving the speed of microscope autofocus and getting rid of the disadvantage of slow focusing speed caused by the traditional need to collect multiple images. In addition, the use of a deep learning model to repair the image and then process the repaired image can better resist the interference caused by complex foregrounds and effectively improve the accuracy and robustness of autofocus. In addition, the present invention uses laser active illumination to generate split-image images rather than ordinary natural images. Therefore, the analysis of fixed features in the image has higher accuracy and robustness, is applicable to more scenarios, and provides new ideas for subsequent autofocus algorithm research and practical applications.

[0149] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the present profession can make slight changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A split-image autofocus method based on deep learning, characterized in that: The steps include: Step S1: collecting the original image at the current position of the microscope; Step S2: extracting the image sub-block where the split-image pattern is located in the original image; Step S3: using the trained deep learning model to repair the image sub-block to obtain a repaired image sub-block; Step S4: Calculating the relative pixel distance between the left and right split-image patterns in the restored image sub-block using a similarity algorithm; Step S5: predicting the defocus distance of the original image from the relative pixel distance according to the pre-calibrated relationship model; Step S6: moving the objective lens to the target position according to the predicted defocus distance to complete the automatic focusing of the microscope; Before repairing the image sub-block using the trained deep learning model, the method includes: Build a deep learning model based on generative adversarial networks; Producing paired data samples for training deep learning models; Training the deep learning model based on the generative adversarial network according to the paired data samples to generate the trained deep learning model; The deep learning model based on the generative adversarial network includes a generator and a discriminator, the generator is based on a deep separable convolutional network, and the discriminator is PatchGAN; The generator first extracts image features by downsampling the received image three times, and then generates a repaired image by upsampling it three times; The discriminator inputs an image, which is downsampled three times and then processed by a fully connected layer, and the Sigmoid function outputs an evaluation value; The step of preparing paired data samples for training a deep learning model includes: The step length and the distance are fixed to collect images of each focal plane under different foregrounds, and the image sub-blocks where the split-image pattern is located in all images are extracted to obtain multiple sets of focal stacks; Determining a reference image sub-block, i.e., an ideal focal plane, for each focus stack according to a judgment criterion, and marking actual defocus distances for other image sub-blocks in each focus stack according to the reference image sub-block; The image sub-blocks marked with the same defocus distance in all focus stacks are grouped into one category; In each type of image sub-block, all clean foreground images are averaged to fit a new image as the label image of each type of image sub-block; Each complex foreground image and the label image corresponding to its class form a pair of data samples to produce all paired data samples; The method of predicting the defocus distance of the original image from the relative pixel distance according to the pre-calibrated relationship model requires calibrating the relationship model between the relative pixel distance and the actual defocus distance in advance, including: Build a relational model based on gradient boosted decision trees; Create a dataset for calibrating the relational model; Calibrate the gradient boosting decision tree-based relationship model according to the data set to generate a calibrated relationship model; The step of preparing a data set for calibrating a relational model includes: The step length and the stroke are fixed to collect images of each focal plane under the clean foreground, and the image sub-blocks where the split-image pattern is located in all images are extracted to obtain multiple sets of focus stacks; The similarity algorithm is used to estimate the relative pixel distances between the left and right split-image patterns in all image sub-blocks; Determine the reference image sub-block of each focus stack, i.e., the ideal focal plane, according to the judgment criterion, and mark the actual defocus distances of other image sub-blocks in each focus stack according to the reference image sub-block; The relative pixel distance and true defocus distance corresponding to each image sub-block are used as a pair of data samples to produce a dataset of all image sub-blocks.

2. The split-image autofocus method based on deep learning according to claim 1, wherein The extracting the image sub-block where the split-image pattern is located in the original image includes: Stretching the contrast of the original image to between 0 and 255; Select a specific threshold T to segment the foreground and background of the original image; The center position of the foreground in the original image is located, and an image sub-block with a size of 1536×1536 is extracted based on the center position of the foreground.

3. The split-image autofocus method based on deep learning according to claim 1, wherein: The step of training the deep learning model based on the generative adversarial network according to the paired data samples to generate the trained deep learning model includes training of a generator and training of a discriminator: The training of the generator includes: Inputting the complex background image in the paired data sample into the generator, outputting the restored image, and inputting the restored image into the discriminator, outputting an evaluation value; A total loss function is obtained according to the root mean square loss of the restored image and the label image and the binary cross entropy loss of the output evaluation value, and the two are weighted in different ratios; Update the generator's network parameters according to the total loss function, and repeat the above process until the generator is trained; The training of the discriminator includes: The label image and the complex background image in the paired data sample are input into the discriminator as true and false images respectively, and the network parameters of the discriminator are updated according to the binary cross entropy loss of the output evaluation value. The above process is repeated until the discriminator is trained.

4. The split-image autofocus method based on deep learning according to claim 1, wherein: The determining of the reference image sub-blocks of each group of focal stacks according to a judgment criterion includes: Using a similarity algorithm to estimate the relative pixel distances between the left and right split-image patterns of all image sub-blocks in each focus stack, selecting the image sub-block with the smallest relative pixel distance as the initial reference image sub-block, and labeling the true defocus distances of the other image sub-blocks in each focus stack according to the initial reference image sub-block; According to the pre-calibrated relationship model, the defocus distance of each image sub-block is predicted by the relative pixel distance. The mean square error between the actual defocus distance and the predicted defocus distance corresponding to the 10 image sub-blocks to the left and right of the initial reference image sub-block is calculated. Select the left and right image sub-blocks of the initial reference image sub-block as new reference image sub-blocks, and calculate their respective mean square errors according to the above steps; Compare the mean square errors of 5 different reference image sub-blocks, and select the image sub-block with the smallest mean square error as the final reference image sub-block.

5. The split-image autofocus method based on deep learning according to claim 1, wherein: The method of calculating the relative pixel distance between the left and right split-image patterns in the restored image sub-block by using a similarity algorithm includes: Segmenting all pixels occupied by the right split-image pattern in the restored image sub-block, calculating the centroid position of the right split-image pattern, and vertically flipping the right split-image pattern; The centroid position of the left split-image pattern is located using a similarity algorithm based on the flipped right split-image pattern; The relative pixel distance between the left and right split-image patterns is obtained according to the centroid positions of the two.

Images