Automatic focusing device and method under nanoscale microscope objective
By combining the autofocus device of convolutional neural network and long and short-term memory network, defocus amount calculation is performed using the correlation information of two adjacent images, which solves the problem that traditional methods cannot effectively solve the defocus amount under a high numerical aperture microscope. The problem of this achieves a more accurate and stable autofocus effect.
Patent Information
- Application Number
- CN202410744838.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2024-11-12
AI Technical Summary
In the existing microscope with high numerical aperture, high resolution, and low depth of field optical systems, traditional calculation factors that characterize the degree of blur cannot effectively calculate the defocus amount, and deep learning methods based on single images are displayed at the nanoscale. The calculation effect under the micro objective lens is poor and stable autofocus cannot be achieved.
The automatic focus device combined with a convolutional neural network and a long and short-term memory network is used to calculate the defocus amount through the correlation information of two adjacent images, and use data set construction, network training and inference modules to achieve a more accurate defocus solution. Compute and autofocus follow.
The defocusing effect and autofocus stability under the nano-scale microscope objective lens are improved, and the problem of poor learning effect of single images is avoided, and fast and efficient autofocus is achieved, which is comparable to active autofocus.
Smart Images

Figure CN118938455A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of microscopic imaging and image processing, and in particular to an automatic focusing device under a nanometer-level microscope objective. Background Art
[0002] In the field of semiconductor or biological detection, in order to obtain higher resolution and better image quality, more advanced microscope objective lenses are often used. These objective lenses have the characteristics of high numerical aperture, low working distance and small depth of field, so the autofocus device is very important in this type of equipment. It is a necessary condition for observation to always keep the sample to be tested in the depth of field.
[0003] The optical system of existing microscope objective lenses is prone to defocusing under high numerical aperture, high resolution, low operating distance and low depth of field objectives. The autofocus methods are mainly divided into active focus and passive focus. The active focus method requires additional equipment to obtain the defocus amount, and the hardware device is often closely related to the optical parameters of the objective lens, which is complex to use and costly. The passive autofocus method uses the image obtained by the sensor to calculate the defocus amount. For the camera autofocus problem in ordinary scenes, the defocus amount can be effectively calculated using an operator that characterizes the degree of blur. However, for nano-scale microscope observations, there are a series of problems in the image such as light source interference, diffraction, and poor light source uniformity. The traditional algorithm directly characterizes the non-uniformity and has large errors, and usually cannot automatically focus. Therefore, it is very necessary to design a reasonable autofocus method.
[0004] The Chinese invention patent with application number CN202310769969.4 provides a natural image autofocus method based on deep learning. It provides a natural image autofocus technology based on deep learning. The implementation of this technology includes several key steps: first, collect natural images from the current microscope position; then, extract regions of interest from the image, which may contain target objects or features of interest; then, use a trained deep learning model to predict the defocus distance of these regions; finally, according to the prediction results, adjust the position of the microscope objective lens so that the image is focused, aiming to solve the problem that microscopes and other devices need to focus manually when observing natural images. The above technology has the following limitations and defects. First, the scope of application of this technology is limited, mainly limited to the focus of natural images, and may not work well in other scenarios, such as medical imaging or industrial detection. Secondly, since the performance of the deep learning model is affected by the training data and model architecture, there may be performance fluctuations in actual applications, resulting in inaccurate focus. In addition, despite the use of hardware acceleration, there may still be delays in the reasoning process in some cases, especially when real-time response is required, which may affect the user experience.
[0005] The Chinese invention patent with application number CN202210674506.5 provides an autofocus method and system based on deep learning, including the following steps: First, obtain a defocused image taken by a camera, and use a trained target deep learning model to calculate the clarity of the image. Then, obtain the step range of the motor, and calculate the number of movement steps of the motor according to the image clarity and the motor step range, and the motor is used to drive the camera to move. Next, control the motor to move for the first time, and then control the motor to move for the second time within a predetermined step range to obtain the target position with the highest clarity in the image of the second movement process. Finally, control the motor to move to the target position to complete the autofocus. The embodiment of the invention uses the predicted value of the network to quickly move the motor to the accurate focus position, thereby greatly improving the speed of autofocus. In addition, the method also reduces the range of the motor search, so that it can show good focusing effect in both extremely dark and multi-light scenes, thereby improving the focusing efficiency. However, although this method has made significant progress in speed and efficiency, there are still limitations, such as further optimization of stability and applicability under extreme conditions. Summary of the invention
[0006] In order to solve the problem that the conventional calculation factors characterizing the degree of image blur often cannot effectively calculate the defocus amount in the existing microscope objective lens in the optical system with high numerical aperture and small depth of field under high resolution proposed in the above background technology, and due to the interference or diffraction effect of light, the deep learning method based on a single image will also have the problem of poor calculation effect. The present invention provides an automatic focusing device under a nanometer-level microscope objective lens, which uses two adjacent defocused images to further utilize the previous image information for network learning, thereby achieving better defocus amount calculation effect and automatic focus following effect.
[0007] To achieve the above-mentioned object, the present invention provides the following technical solutions: an automatic focusing device under a nano-level microscope objective, comprising a data set construction unit, an image defocus amount calculation unit and a displacement motion platform;
[0008] The data set construction unit is used to collect and process different image data, and the data set construction unit includes a data acquisition module and a data processing module;
[0009] The image defocus calculation unit is used to calculate the correlation information of adjacent images to solve the corresponding defocus amount and analyze the movement direction of the motion platform; the image defocus calculation unit includes a deep learning network structure construction module, a network training module and a network reasoning module, wherein the deep learning network structure construction module includes a convolutional neural network model and a long short-term memory network model;
[0010] The displacement motion platform is used to drive the displacement of the objective lens.
[0011] The network training module includes a calculation loop, an optimization loop and a weight parameter loop.
[0012] The data acquisition module is used to collect defocused images, and to obtain the front defocused images and the back defocused images respectively; the data processing module is used to decompose the two different images taken and then to splice them;
[0013] The convolutional neural network model is used to abstractly express and extract features from the input tensor; the long short-term memory network model is used to further learn the temporal correlation of the feature data extracted by the convolutional neural network model, and finally output a single data volume for each time series, and achieve the regression of the defocus amount of two sets of adjacent images through the final fully connected network; the network training module is used to learn and train the features and patterns of the input data, so as to improve its performance in various tasks;
[0014] The calculation loop is used to forward calculate the output value of each neuron. The optimization loop is used to determine the optimization objective function. The weight parameter loop is used to update the network weight parameters according to the gradient guidance of the loss function;
[0015] The network reasoning module is used to use the previous correlation information of the image to calculate the corresponding defocus amount and analyze the movement direction of the motion platform.
[0016] A method for automatic focusing under a nano-level microscope objective lens, which utilizes the above-mentioned automatic focusing device under a nano-level microscope objective lens to achieve nano-level microscope objective lens focusing, comprises the following steps:
[0017] S1 Dataset Construction
[0018] Using the data acquisition module of the data set construction unit to acquire front defocus images and rear defocus images respectively; using the data processing module to decompose and splice a large number of defocus images at different positions acquired by the data acquisition module to obtain a data set, and input the data set into a long short-term memory network model;
[0019] S2 deep learning network structure construction
[0020] A convolutional neural network model is established based on the mainstream convolutional neural network skeleton. The input tensor is used for abstract expression and extraction of features. The feature data extracted by the convolutional neural network model is sent to the long short-term memory network model for further time series correlation learning. Finally, a single data volume is output for each time series, and the defocus regression of two sets of adjacent images is realized through the final fully connected network.
[0021] S3 Network Training
[0022] The network training module is used to learn and train the characteristics and patterns of the input data to obtain a training set. First, the calculation loop is used to forward calculate the output value of each neuron, and then the optimization loop is used to determine the optimization objective function. Then, the weight parameter loop is used to update the network weight parameters according to the gradient guidance of the loss function.
[0023] S4 Network Inference
[0024] The two adjacent images are sent to the network reasoning module, and the image association information is obtained by using the training set obtained after training the training network module. The corresponding defocus amount is calculated through the image association information, and the movement direction of the motion platform is analyzed;
[0025] S5 adjusts the focus of the objective lens
[0026] After calculating two accurate defocus amounts through network reasoning, the correctness of the motion platform's movement direction is analyzed, and the objective lens is displaced accordingly by displacing the motion platform to achieve automatic focusing.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] The present invention utilizes a convolutional neural network model in combination with a long short-term memory network model to calculate the defocus amount of two adjacent images and uses serialized data, which can further utilize previous image information for network learning, thereby achieving better defocus amount calculation effect and automatic focus following effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 The automatic focusing device under the nanometer-level microscope objective provided by the present invention is a schematic diagram;
[0030] Figure 2 It is a diagram of image decomposition by the data processing module in the present invention;
[0031] Figure 3 Schematic diagram of the deep learning network structure in the present invention; Figure 4 Flow chart of the automatic focusing method provided by the present invention.
[0032] In the accompanying drawings shown, there are an autofocus device 100, a data set construction unit 110, a data acquisition module 111, a data processing module 112, an image defocus calculation unit 120, a deep learning network structure construction module 121, a convolutional neural network model 1211, a long short-term memory network model 1212, a network training module 122, a calculation loop 1221, an optimization loop 1222, a weight parameter loop 1223, a network reasoning module 123, a displacement motion platform 130, and an objective lens 140. DETAILED DESCRIPTION
[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0034] like Figure 1 As shown, a nano-level microscopic automatic focusing device 100 includes a data set construction unit 110, an image defocus amount calculation unit 120 and a displacement motion platform 130;
[0035] It includes a data set construction unit 110 for collecting and processing different image data, and the data set construction unit 110 includes a data acquisition module 111 and a data processing module 112;
[0036] The data acquisition module 111 is used to collect defocused images, and respectively acquire the front defocused images and the rear defocused images; the data processing module 112 is used to decompose the two different images taken, and then splice them;
[0037] After manually focusing, the defocused images are collected by the data acquisition module 111. The front defocused images and the back defocused images are collected respectively. After collecting a large number of defocused images at different positions, the images are decomposed by the data processing module 112 to ensure that the recurrent neural network can be used. Figure 2 As shown, the decomposition process is to splice two adjacent defocused images of size into one image, and then split the spliced image into 2n n-dimensional vectors by row. These vectors can be regarded as 2n interrelated time-series inputs, and these split vectors are input into the image defocus calculation unit 120 in sequence.
[0038] The image defocus amount calculation unit 120 is used to calculate the adjacent image association information to solve the corresponding defocus amount and analyze the motion direction of the motion platform; the image defocus amount calculation unit 120 includes a deep learning network structure construction module 121, a network training module 122 and a network reasoning module 123;
[0039] The deep learning network structure construction module 121 includes a convolutional neural network model 1211 and a long short-term memory network model 1212; the deep learning network structure construction module 121 is based on a convolutional neural network (CNN) in conjunction with a long short-term memory network (LSTM), and the overall structure of the deep learning network structure construction module 121 is as follows: Figure 3As shown, the input of the deep learning network structure building module 121 is 2n interrelated time series inputs, each of which is a tensor with a dimension size of 1×n. The input tensor first enters the convolutional neural network (CNN) for abstract expression and extraction of features. In the present invention, the CNN network is mainly based on the mainstream convolutional neural network skeleton, such as the residual network (ResNet). The feature data extracted by CNN is sent to the long short-term memory network (LSTM) for further time series correlation learning. Finally, each time series outputs a single data amount, and the defocus regression of two sets of adjacent images is realized through the final fully connected network.
[0040] The network training module 122 includes a calculation loop 1221, an optimization loop 1222 and a weight parameter loop 1223;
[0041] The calculation loop 1221 is used to forward calculate the output value of each neuron. For LSTM, the corresponding neuron values are calculated separately. Specifically, each neuron in the LSTM network includes an input gate, a forget gate, an output gate, and a cell state. In the forward calculation, the values of the input gate, the forget gate, and the output gate are first calculated. These gates control the flow and retention of information. Then, the cell state is updated based on the output of these gates and the current input. Finally, the output value at the current moment is calculated using the cell state and the output of the output gate.
[0042] The optimization loop 1222 is used to determine the optimization objective function. In the early stage of training, the output value and the expected value will be inconsistent, so the error term value of each neuron is calculated to construct a loss function. By comparing the output of the network with the expected output, the error of each time step can be calculated. Finally, the regression of the defocus amount is realized through full connection in the network. At this time, the mean square error (MSE) is used as the loss function, that is:
[0043]
[0044] Where N represents the total number of output defocus values during training, y i Indicates the actual defocus amount, Δ i Represents the defocus data output by the network.
[0045] The weight parameter loop 1223 is used to update the network weight parameters according to the gradient guidance of the loss function. Similar to the traditional recurrent neural network (RNN), the back propagation of the LSTM error term includes two levels: one is the spatial level, which propagates the error term to the previous layer of the network. The other is the temporal level, which is back propagated along time, that is, starting from the current moment t, the error at each moment is calculated. At each time step, the gradient of the loss function to the parameter is calculated by the chain rule, and then the optimization algorithm such as gradient descent is used to update the parameters to minimize the loss function. In this way, the network can gradually learn the characteristics and patterns of the input data, thereby improving its performance in various tasks.
[0046] The network reasoning module 123 is used to use the previous correlation information of the images to calculate the corresponding defocus amount and analyze the movement direction of the motion platform. After the network is trained using a training set containing sufficient data, the network can be fed with two adjacent images and make full use of the previous correlation information of the images to calculate the corresponding defocus amount.
[0047] The displacement motion platform 130 is used to drive the displacement of the objective lens 140. After calculating two accurate defocus amounts through the above-mentioned network reasoning module 123, the correctness of the movement direction of the motion platform can be effectively analyzed. After starting the control system of the displacement motion platform 130 to perform corresponding displacement on the objective lens 140, the final autofocus process can be realized.
[0048] like Figure 4 As shown, a method for automatic focusing under a nano-level microscope objective lens is provided. The method uses the above-mentioned automatic focusing device 100 under a nano-level microscope objective lens to achieve nano-level microscope objective lens focusing, and comprises the following steps:
[0049] S1 Dataset Construction
[0050] The data acquisition module 111 of the data set construction unit 110 is used to acquire the front defocused images and the rear defocused images respectively; the data processing module 112 is used to decompose and splice a large number of defocused images at different positions acquired by the data acquisition module 111 to obtain a data set, and the data set is input into the long short-term memory network model 1212;
[0051] S2 deep learning network structure construction
[0052] A convolutional neural network model 1211 is established based on the mainstream convolutional neural network skeleton, and the input tensor is abstractly expressed and extracted for features. The feature data extracted by the convolutional neural network model 1211 is sent to the long short-term memory network model 1212 for further time series correlation learning. Finally, a single data amount is output for each time series, and the defocus amount regression of two sets of adjacent images is realized through the final fully connected network.
[0053] S3 Network Training
[0054] The network training module 122 is used to learn and train the characteristics and patterns of the input data to obtain a training set. First, the calculation loop 1221 is used to forward calculate the output value of each neuron, and then the optimization loop 1222 is used to determine the optimization objective function, and then the weight parameter loop 1223 is used to update the network weight parameters according to the gradient guidance of the loss function;
[0055] S4 Network Inference
[0056] Send two adjacent images to the network reasoning module 123, use the training set obtained after training the training network module to obtain image association information, calculate the corresponding defocus amount through the image association information, and analyze the movement direction of the motion platform;
[0057] S5 adjusts the focus of the objective lens 140
[0058] After calculating two accurate defocus amounts through network reasoning, the correctness of the motion direction of the motion platform is analyzed, and the objective lens 140 is displaced accordingly by displacing the motion platform 130 to achieve automatic focusing.
[0059] The present invention proposes an automatic focusing method under a nano-scale microscope objective lens based on a long short-term memory network. By constructing and training the corresponding network, the change regression value of two adjacent images can be output during the network forward reasoning, and the movement direction of the motion platform can be further guided by the change score until the change regression value tends to be stable and is less than the set threshold to achieve full focus. The proposed method evaluates the displacement direction and step size by learning the difference between two adjacent images with different defocus amounts, effectively avoiding the problem of poor learning effect of a single image. And because the network structure is simple and cooperates with a reasonable graphics processor, the speed of automatic focusing is very fast, which can be comparable to the effect of active automatic focusing in many sample movement scenarios.
Claims
1. An automatic focusing device under a nano-level microscope objective, characterized in that: It includes a data set construction unit, an image defocus calculation unit and a displacement motion platform; The data set construction unit is used to collect and process different image data, and the data set construction unit includes a data acquisition module and a data processing module; The image defocus calculation unit is used to calculate the correlation information of adjacent images to solve the corresponding defocus amount and analyze the movement direction of the motion platform; the image defocus calculation unit includes a deep learning network structure construction module, a network training module and a network reasoning module, wherein the deep learning network structure construction module includes a convolutional neural network model and a long short-term memory network model; The displacement motion platform is used to drive the displacement of the objective lens.
2. The nanometer-level microscopic automatic focusing device according to claim 1, characterized in that: The network training module includes a calculation loop, an optimization loop and a weight parameter loop.
3. The nanometer-level microscopic automatic focusing device according to claim 1, characterized in that: The data acquisition module is used to collect defocused images, and to obtain the front defocused images and the back defocused images respectively; the data processing module is used to decompose the two different images taken and then to splice them; The convolutional neural network model is used to abstractly express and extract features from the input tensor; the long short-term memory network model is used to further learn the temporal correlation of the feature data extracted by the convolutional neural network model, and finally output a single data volume for each time series, and achieve the regression of the defocus amount of two sets of adjacent images through the final fully connected network; the network training module is used to learn and train the features and patterns of the input data, so as to improve its performance in various tasks; The calculation loop is used to forward calculate the output value of each neuron. The optimization loop is used to determine the optimization objective function. The weight parameter loop is used to update the network weight parameters according to the gradient guidance of the loss function; The network reasoning module is used to use the previous correlation information of the image to calculate the corresponding defocus amount and analyze the movement direction of the motion platform.
4. A method for automatic focusing under a nano-level microscope objective lens, the method utilizing the automatic focusing device under a nano-level microscope objective lens in the above claims 1-3 to realize nano-level microscope objective lens focusing, characterized in that: The following steps are involved: S1 Dataset Construction Using the data acquisition module of the data set construction unit to acquire front defocus images and rear defocus images respectively; using the data processing module to decompose and splice a large number of defocus images at different positions acquired by the data acquisition module to obtain a data set, and input the data set into a long short-term memory network model; S2 deep learning network structure construction A convolutional neural network model is established based on the mainstream convolutional neural network skeleton. The input tensor is used for abstract expression and extraction of features. The feature data extracted by the convolutional neural network model is sent to the long short-term memory network model for further time series correlation learning. Finally, a single data volume is output for each time series, and the defocus regression of two sets of adjacent images is realized through the final fully connected network. S3 Network Training The network training module is used to learn and train the characteristics and patterns of the input data to obtain a training set. First, the calculation loop is used to forward calculate the output value of each neuron, and then the optimization loop is used to determine the optimization objective function. Then, the weight parameter loop is used to update the network weight parameters according to the gradient guidance of the loss function. S4 Network Inference The two adjacent images are sent to the network reasoning module, and the image association information is obtained by using the training set obtained after training the training network module. The corresponding defocus amount is calculated through the image association information, and the movement direction of the motion platform is analyzed; S5 adjusts the focus of the objective lens After calculating two accurate defocus amounts through network reasoning, the correctness of the motion platform's movement direction is analyzed, and the objective lens is displaced accordingly by displacing the motion platform to achieve automatic focusing.
