Microscope imaging definition detection method based on threshold selection

Abstract

The invention discloses a microscope imaging definition detection method based on threshold selection, and relates to the technical field of computer vision and image recognition. Comprising the following steps: S1, synchronizing field areas and magnification times of an electronic eyepiece and an optical eyepiece; s2, constructing a discrete coding space of an acquisition condition in an electronic eyepiece image acquisition process; s3, synchronous observation and definition detection of the electronic eyepiece and the optical eyepiece based on the discretized coding space parameter condition; and S4, a variable distribution sample microscope imaging definition judgment method of the time sequence image. According to the invention, definition detection errors caused by field-of-view difference between the electronic eyepiece and the optical eyepiece are reduced; image definition evaluation can be conveniently carried out on different microscopic sample images under the same condition, and a reference quantitative basis is provided for definition judgment of microscopic images under different quantitative conditions; and an overall evaluation strategy based on clustering is given for the condition that the definition is not uniformly distributed in the microscopic image region.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image recognition, and specifically to a method for detecting the clarity of microscope imaging based on threshold selection. Background Art

[0002] Microscope imaging can help people observe and analyze the spatial structure attributes and even temporal characteristics of substances in the microscopic environment at different visual scales. By adjusting its various components, the microscope can achieve a continuous transformation from presenting a blurred state of the sample to an enlarged state with clear tissue edge contours and internal structures. Traditional optical eyepieces are convenient for people to observe microscopic results, but they do not have the functions of data quantification storage and backup, and cannot be recognized, processed, and analyzed through technologies such as image processing, and the presented results cannot be reviewed after the microscope is adjusted. With the rapid development and wide application of hardware levels such as deep learning, artificial intelligence, image processing technology, graphics acceleration graphics cards, and electronic eyepieces, data storage, processing, and analysis under microscopic conditions have become possible.

[0003] An electronic eyepiece uses an electron beam instead of light to form a magnified image of the sample. Compared with traditional optical eyepieces, when an electronic eyepiece is installed, the microscope can obtain spatial information at a smaller scale and a display result with a higher magnification, and can observe finer structures and higher clarity details. It is widely used in fields such as scientific research, materials science, biology, medicine, and the semiconductor industry. However, this also makes there be significant differences in the field of view size and spatial scale between the images obtained by the electronic eyepiece and the results observed by the observer through the optical eyepiece. Even for a trinocular microscope with a high price, after directly installing an electronic eyepiece, it often can only ensure that the information seen by the optical eyepiece and the image of the electronic eyepiece change synchronously in clarity, and cannot ensure that the electronic eyepiece has the same or even larger field of view as the optical eyepiece.

[0004] Without considering the field of view difference, by appropriately selecting the electronic eyepiece, it can be made to change synchronously in clarity with the results obtained by the optical eyepiece. That is, under the above conditions, to judge whether the image obtained by the optical eyepiece is clear, it can be achieved by calculating the clarity of the electronic eyepiece image. And under the condition of the field of view brightness where the image area is not overexposed and the light source intensity is fixed, for a fixed observation sample, the clarity of the microscopic image obtained by the electronic eyepiece can be judged by using the method of solving the image clarity in image processing. By calculating image parameters (such as variance, gradient, Laplacian image mean, etc.) based on image grayscale or specific spectra, and selecting an appropriate threshold or threshold function to perform thresholding processing to judge the image clarity: when the calculated image parameter is larger (or smaller), the image can be regarded as being in a clear state; otherwise, the image is in a blurred state. Among them, the threshold function may be a linear, non-linear, or function without an analytical representation.

[0005] When calculating the above image sharpness parameters, it is crucial to select an appropriate field brightness. There is a certain range for selecting the field brightness suitable for observing a microscope, and it is not a fixed value. Within a fixed field range, if the field brightness is too high, it is likely to cause discomfort to the observer's eyes or overexposure of the electronic eyepiece. If the field brightness is too low, the presentation of the detailed structure of the object may be insufficient, and even the magnified image result may not be observable. Under different microscopic conditions, such as observing different objects or using objective lenses with different magnification factors, in order to make the results observed by the optical eyepiece and the electronic eyepiece tend to be consistent, it is necessary to appropriately change the field brightness to adapt to the imaging requirements.

[0006] The adjustment of the field brightness can generally be completed by adjusting the light flux of the microscope (the intensity of the LED light source or the intensity of the natural light reflected towards the objective lens) and selecting different aperture sizes. Adjusting the exposure duration of the electronic eyepiece, changing the objective lenses and eyepieces with different magnification factors in the light path, and performing image gray-scale enhancement processing on the imaged image can also be approximately equivalent to changing the field brightness. Among them, adjusting the light flux of the microscope is the main factor that is difficult to quantitatively evaluate numerically: Under natural light source conditions, the field brightness of the microscope is greatly affected by the intensity of natural environmental light and the adjustment of the reflector angle, and usually no quantitative and functional processing is performed; In order to adapt to different sample microscopic displays and simulate natural light sources, the adjustment of artificial light sources needs to be continuously changed and often there is no digital quantification process. Methods such as adaptive exposure duration and image gray-scale enhancement processing often result in significant differences between the images formed by the electronic eyepiece and the results seen by the human eye through the optical eyepiece. Since the field brightness directly affects the calculation result of the sharpness parameter, a single threshold cannot guarantee the sharpness judgment of the images obtained by the electronic eyepiece.

[0007] In addition, due to the influence of uncertain factors during the sample production process, such as the amount of sample added or the differences in the type, concentration, and distribution of staining agents during the production of permanent human blood smears and temporary yeast slides, even under the same observation conditions such as ensuring the same field brightness, it is impossible to use a fixed threshold to evaluate the sharpness of image parameters.

[0008] In summary, due to the differences between the results of the electronic eyepiece and the optical eyepiece, the changes in the light source brightness, the adjustment of the field brightness, the selection of objective lenses, the differences in the structural characteristics of the observed samples, and many other factors, it is difficult to directly use the image sharpness parameters to judge the sharpness of the results presented by the optical eyepiece with the existing methods. Summary of the Invention

[0009] In order to solve the problem of self-adaptive judgment of the sharpness of the imaging results of the optical eyepiece when the current microscope images under different conditions such as different field brightnesses, the present invention provides a method for detecting the sharpness of microscope imaging based on threshold selection, which is based on microscopic condition quantization coding and threshold selection.

[0010] The present invention is realized through the following technical solutions: A method for detecting the clarity of microscope imaging based on threshold selection, comprising the following steps:

[0011] S1: Synchronization processing of the field of view area and magnification of the electronic eyepiece and the optical eyepiece;

[0012] S2: Construction of a discretized coding space for the acquisition conditions during the process of obtaining the electronic eyepiece image;

[0013] S3: Synchronous observation and clarity detection of the electronic eyepiece and the optical eyepiece based on the parameters of the discretized coding space;

[0014] S4: A method for judging the clarity of microscope imaging of variable-distribution sample time-series images.

[0015] The detailed content corresponding to each step is as follows:

[0016] The specific content of step S1 is as follows: By using an electronic eyepiece, an eyepiece end reducing lens, and a combined extension tube, the synchronous field of view area and clarity change between the electronic eyepiece and the optical eyepiece are realized;

[0017] The specific content of step S2 is as follows: According to different magnification objectives and structurally similar microscopic samples, the maximum field of view intensity allowed for microscopic imaging and the longest exposure duration of the electronic eyepiece are manually determined synchronously, and together with the objective magnification and sample category, microscopic condition coding is performed to form a coding space;

[0018] The specific content of step S3 is as follows: For fixed microscopic conditions, a functional fitting method is used to determine the functional relationship between the light intensity change and the clarity coefficient of the image obtained by the electronic eyepiece, and a clarity threshold function is determined according to manual annotation to realize the clarity detection of the observation result of the optical eyepiece;

[0019] The specific content of step S4 is as follows: For a variable-distribution sample object to be observed, where a variable-distribution sample refers to an object to be observed whose sample not only changes but also has uneven distribution in different regions, clustering analysis is performed using the high-frequency features of the microscopic image pixels, including gradient vectors and Laplace transforms. The two categories with the largest number of pixels included in the category are compared, threshold segmentation is performed according to the within-class mean, and the intersection-over-union ratio is calculated with the segmentation result of the same threshold at a previous selected time point, and then the overall clarity of the image is judged by thresholding.

[0020] The present invention proposes a method for detecting the clarity of microscope imaging based on monitoring the change of field of view brightness, which is used to overcome the threshold selection strategy for the change of field of view brightness. By constructing a quantization coding space for the microscope acquisition conditions and performing clarity detection of the electronic eyepiece image on the basis of the selected coding coordinates, it is used to synchronously evaluate the clarity of the observation content of the microscope optical eyepiece, and can be applied to fields such as microscope imaging, computer vision, and video analysis technology.

[0021] Compared with the prior art, the present invention has the following beneficial effects: A method for detecting the clarity of microscope imaging based on threshold selection provided by the present invention: 1) Clarity evaluation is carried out based on the approximate field of view range of the electronic eyepiece and the optical eyepiece, reducing the clarity detection error caused by the field of view difference between the electronic eyepiece and the optical eyepiece; 2) The microscope imaging conditions are quantitatively represented by using the discrete coding method, which is convenient for evaluating the image clarity of different microscopic sample images under the same conditions, and also provides a reference quantitative basis for judging the clarity of microscopic images under different quantitative conditions; 3) It realizes the real-time and synchronous recording of the observation results of the optical eyepiece and the clarity evaluation basis by using the electronic eyepiece, and realizes the storage and backup of microscope observation data; 4) A global evaluation strategy based on clustering is given for the situation where the clarity is unevenly distributed in the microscopic image area. Specific embodiments

[0022] The present invention will be further described below in conjunction with specific embodiments.

[0023] A method for detecting the clarity of microscope imaging based on threshold selection includes the following steps:

[0024] S1: Synchronization processing of the field of view area and magnification of the electronic eyepiece and the optical eyepiece, specifically as follows:

[0025] Select an electronic eyepiece with a spatial resolution higher than 2 million pixels and an imaging unit area not less than 1 / 2.5 inches. Using its automatic exposure function in combination with a reduction lens with an extension tube (effective extension length l), the electronic eyepiece image and the microscopic image of the optical eyepiece have a similar field of view and the same horizontal visual effect, ensuring that clear samples can be observed synchronously under different magnification objectives.

[0026] S2: Construction of a discrete coding space for the acquisition conditions in the process of obtaining the electronic eyepiece image, specifically as follows:

[0027] For a group of magnification objectives and aperture combinations, under the condition of the maximum light flux φ max (when the artificial light source intensity is the maximum or the natural light source in the experimental site is the strongest), manually observe and select the longest exposure duration of the electronic eyepiece that makes the clear image of the electronic eyepiece without obvious overexposure And set the actual exposure duration of the electronic eyepiece as At this time, the pixel gray value of the image obtained by the electronic eyepiece can change with the light source intensity, where N O represents the objective magnification, ΔT≥0 and Encode combinations of different magnification objectives, aperture types, and electronic eyepiece exposure durations to form an encoding set C, and ensure that the elements C in C iThere is a unique one-to-one correspondence with the combination of the objective lens category, the aperture category, and the corresponding exposure duration of the electronic eyepiece.

[0028] S3: Synchronous observation and clarity detection of the electronic eyepiece and the optical eyepiece under the condition of discretized encoded space parameters; specifically as follows:

[0029] 3.1) Keep the light flux unchanged and the sample can be clearly presented in the microscope field of view. Under a certain determined C i condition, obtain a set J of clear images obtained by the electronic eyepiece with n images by moving the sample position and replacing the same type of sample.

[0030] 3.2) Perform grayscale processing (or spectral selection), smoothing filtering, and sharpening filtering operations on each image in the image set J in turn. The height and width of J are M and N pixels respectively, and I represents the grayscale image of any image in the image set J. Smoothing filtering is used to eliminate the possible noise signals in the microscopic image of the electronic eyepiece, and a mean filter, Gaussian smoothing filter, median filter, etc. can be selected; sharpening filtering obtains the high-frequency information corresponding to the corners, edge lines, etc. in the clear image. The filter can be a first-order gradient operator such as the Sobel operator, Prewitt operator, etc. with a window size of k, or a second-order gradient operator such as the Laplacian operator, Laplacian of Gaussian (LoG) operator, etc., or other methods that can obtain the high-frequency information of the image, such as the multi-scale method of the high-frequency component of wavelet transform and other tools. For the first-order gradient operator, taking the filtering result of the Sobel operator as an example, assuming that the gradient images obtained in the horizontal and vertical directions are dx and dy respectively, the clarity parameter S of the overall image can be calculated according to the following formula:

[0031] S = c·f(dx,dy) (1)

[0032] c is a constant coefficient; f(dx,dy) can be a function relationship based on the 2-norm or other higher-order norms as shown in formula (2) that can convert a vector into a single non-negative scalar:

[0033]

[0034] 3.3) For a specific type of sample with a stable microscopic structure (such as onion scale leaves, plant root tips, etc., the clarity degree in the microscope eyepiece field of view is basically the same), under the condition that other observation conditions remain unchanged (without changing the microscopic condition C i ), continuously change the microscope light flux φ in the range of [0,φ max , and record all J iThe gray-scale average value (such as formula (3)) or the average energy value (such as formula (4)), etc., the value I that can reflect the change in the field brightness V SP and S SP The parameters form a two-dimensional pair in the form of (I SP , S SP ).

[0035]

[0036]

[0037]

[0038] Among them, I ij is the gray value of the image pixel.

[0039] 3.4) Only by rotating the coarse / fine focusing screw, obtain the set of non-clear microscope electronic eyepiece images under the condition of C i Repeat steps 3.1) and 3.2) to obtain multiple groups of (I sp , S sp ) two-dimensional pairs in different non-clear situations.

[0040] 3.5) For the (I i , S SP ) and (I sp , S sp ) obtained under the same C i condition, construct the union U, and screen the subset according to the same I component in the set element (I, S) Generate a two-dimensional feature vector (I i , (mS min +nS max ) / (m + n)) ∈ V from the S components of the elements of U min , where m, n ∈ R+, and S max and S S need to satisfy formula (5), and T i is the preset difference threshold.

[0041] 3.6) Perform polynomial function fitting according to the set V to obtain the threshold generation function Th = F(I). Under the condition of C new , calculate Th new from the (I new , S new ) obtained by calculating according to the threshold generation function F and the microscope electronic eyepiece image. When S new > Th i then the image is clear; otherwise the imaging is not clear.

[0042] S4: Method for Judging the Imaging Clarity of Variable Distribution Samples in Sequential Images

[0043] 4.1) For samples with a stable microscopic structure but uneven distribution within the microscope field of view (such as a temporary slide of yeast solution, etc.), under microscopic condition C i obtain the gradient images dx and dy in the horizontal and vertical directions of the grayscale image, and the clarity parameter S using the method in step 3.2). Use unsupervised learning (such as Kmeans clustering) to perform clustering analysis on the clarity parameter S of local image regions (b×b pixels) in the microscopic image. S b is calculated as shown in formula (6). Select the class centers SC1 and SC2 of the top two categories with the largest number of class samples, where SC1>SC2, and use b as the threshold to segment the clarity parameter S to obtain the segmentation result SS , where f and respectively represent the actual average block clarity parameters of the two types of regions.

[0044]

[0045] 4.2) Repeat step 4.1) after N t frames under the same acquisition conditions to obtain a new segmentation result SS c and calculate the intersection-over-union ratio between the segmentation result SS c and SS f . When is greater than Th , it is judged that the overall microscopic imaging is clear; otherwise, it is considered unclear. IOU

[0046] The scope of protection required by the present invention is not limited to the above specific embodiments. Moreover, for those skilled in the art, the present invention can have various deformations and modifications. Any modifications, improvements, and equivalent replacements made within the concept and principle of the present invention should be included within the protection scope of the present invention.​

Claims

1. A method for detecting the clarity of microscope imaging based on threshold selection, characterized in that: The steps include: S1: Synchronization of the field of view and magnification of the electronic eyepiece and the optical eyepiece; S2: Construction of the discretized coding space of acquisition conditions during the electronic eyepiece image acquisition process; S3: Synchronous observation and clarity detection of electronic eyepiece and optical eyepiece based on discrete coding space parameters; S4: Method for judging the imaging clarity of variable distribution samples in time-series images using microscope.

2. The method for detecting the imaging clarity of a microscope based on threshold selection according to claim 1, characterized in that: The steps include: The step S1 specifically includes: using the electronic eyepiece, the eyepiece end zoom lens and the combined extension tube to achieve synchronous field of view and clarity changes of the electronic eyepiece and the optical eyepiece; The step S2 specifically includes: manually determining the maximum field intensity allowed for microscopic imaging and the longest exposure time of the electronic eyepiece according to different magnification objective lenses and microscopic samples with similar structures, and encoding the microscopic conditions together with the objective lens magnification and the sample category to form a coding space; The step S3 specifically comprises: for fixed microscopic conditions, a function fitting method is used to determine the functional relationship between the light intensity change and the clarity coefficient of the image obtained by the electronic eyepiece, and a clarity threshold function is determined according to manual annotation to achieve clarity detection of the optical eyepiece observation result; The step S4 is specifically as follows: for the variable distribution sample to be observed, cluster analysis is performed using the high-frequency features of the microscopic image pixels, including gradient vectors and Laplace transforms, the two categories with the largest number of pixels are compared, threshold segmentation is performed according to the intra-class mean, and the category intersection-union ratio is calculated compared with the segmentation result of the same threshold at a previous time point, and then the overall clarity of the image is determined by thresholding.

3. The method for detecting the clarity of microscope imaging based on threshold selection according to claim 2, wherein: Step S1 is specifically as follows: select an electronic eyepiece with a spatial resolution higher than 2 million pixels and an imaging unit area of not less than 1 / 2.5 inch, utilize its automatic exposure function in combination with a zoom lens with an extension tube, wherein the effective extension length of the extension tube is l, so as to achieve a similar field of view and the same level of visual effect between the electronic eyepiece image and the optical eyepiece microscopic image, and ensure that clear samples can be observed synchronously under different magnification objective lens conditions.

4. A method for detecting the clarity of microscope imaging based on threshold selection according to claim 3, characterized in that: Step S2 is specifically as follows: For a set of objective lens magnification and aperture combinations, at the maximum light flux φ max that is, under the condition of the maximum artificial light source intensity or the strongest natural light source in the experimental site, manually observe and select the longest exposure duration of the electronic eyepiece that makes the clear image of the electronic eyepiece have no obvious overexposure and set the actual exposure duration of the electronic eyepiece to be At this time, the pixel gray value of the image obtained by the electronic eyepiece changes with the light source intensity, where N o represents the objective lens magnification, ΔT≥0 and Encode combinations of objective lenses with different magnifications, aperture types, and electronic eyepiece exposure durations to form an encoding set C, ensuring that the element C in C i has a unique one-to-one correspondence with the combination of the objective lens category, aperture category, and the corresponding electronic eyepiece exposure duration.

5. A method for detecting the clarity of microscope imaging based on threshold selection according to claim 4, characterized in that: Step S3 is as follows: 3.1) Keeping the luminous flux unchanged and the sample can be clearly presented in the microscope field of view, under a certain determined C i condition, obtain a set J of clear images obtained by the electronic eyepiece through moving the sample position and replacing the same type of samples to obtain n images. 3.2) grayscale processing, smoothing filtering and sharpening filtering operations are sequentially performed on each image in the image set J, wherein the grayscale processing may be spectral selection; wherein the height and width of J are M and N pixels respectively, and I represents the grayscale image of any image in the image set J; smoothing filtering is used to eliminate noise signals appearing in the electronic eyepiece microscopic image, and a mean filter, a Gaussian smoothing filter or a median filter is selected; The sharpening filter obtains the corresponding high-frequency information in the clear image. The filter selects a first-order gradient operator or a second-order gradient operator with a window size of k. The first-order gradient operator is a Sobel operator. Assuming that the horizontal and vertical gradient images are dx and dy respectively, the clarity parameter S of the overall image is calculated according to the following formula: S=c·f(dx,dy) (1) Where c is a constant coefficient, and f(dx,dy) is a function relationship based on the 2-norm or higher-order norm as shown in formula (2) that converts a vector into a single non-negative scalar; 3.3) For a specific category of samples with a stable microstructure, under the condition that other observation conditions remain unchanged, including without changing the microscopic condition C i , continuously vary the microscope light flux φ within the range of [0, φ max , and record the average gray value of all J i , such as in formula (3), or the average energy value, such as in formula (4), and the value I SP reflecting the change in the field brightness V, as well as S SP . The parameters form a two-dimensional pair in the form of (I SP , S SP ); Among them, I ij is the gray value of the image pixel; 3.4) Obtain a set of non-clear microscope electronic eyepiece images of the microscope under condition C only by rotating the coarse / fine focusing screw, and repeat steps 3.1) and 3.2) to obtain multiple groups (I i under different non-clear conditions, and obtain a two-dimensional pair of numbers (I sp , S sp ); 3.5) For the same C i Under the conditions obtained (I SP ,S SP ) and (I sp ,S sp ) constructs a union U and selects subsets based on the same I component in the set elements (I, S) byU i The S component of the element generates a two-dimensional eigenvector (I i ,(mS min +nS max ) / (m+n))∈V, where m,n∈R+, and S min and S max Satisfying formula (5), T S is the preset difference threshold; 3.6) Perform polynomial function fitting according to the set V, that is, obtain the threshold generating function Th = F(I); in C i Under these conditions, the threshold generation function F and the microscope electron eyepiece image are calculated to obtain (I new ,S new ) to obtain Th new , when S new >Th new Otherwise, the image is not clear.

6. The method for detecting the clarity of microscope imaging based on threshold selection according to claim 5, characterized in that: Step S4 is specifically as follows: 4.1) For samples with stable microstructure but uneven distribution within the microscope field of view, under microscopic condition C i Step 3.2) is used to obtain the horizontal and vertical gradient images dx and dy of the grayscale image, and the clarity parameter S; the clarity parameter S of the local image area in the microscopic image is calculated by unsupervised learning. b Perform cluster analysis, the local image area is b×b pixels, S b Calculate as formula (6); select the class centers SC1 and SC2 of the first two classes with the largest number of class samples, SC1>SC2, and As the threshold segmentation clarity parameter S, the segmentation result SS is obtained. f ,in and Represent the actual average block clarity parameters of the two types of areas; 4.2) Repeat step 4.1) after N frames under the same acquisition conditions to obtain a new segmentation result SS t c And calculate the segmentation result SS c And SS f The intersection over union between them When Greater than Th IOU It is judged that the overall microscopic imaging is clear, otherwise it is considered unclear.​ 7. A method for detecting the clarity of microscope imaging based on threshold selection according to claim 5, characterized in that: In step 3.2), the high-frequency information includes corner points and edge line information; the first-order gradient operators include the Sobel operator and the Prewitt operator; the second-order gradient operators include the Laplacian operator and the Laplacian of Gaussian (LoG) operator.

8. A method for detecting the clarity of microscope imaging based on threshold selection according to claim 5, characterized in that: In step 3.3), the specific category of samples with stable microscopic structures are samples with basically the same clarity within the field of view of the microscope eyepiece, including, for example, onion scale leaves and plant root tips.

9. A method for detecting the clarity of microscope imaging based on threshold selection according to claim 6, characterized in that: In step 4.1), the unsupervised learning is Kmeans clustering.

10. A method for detecting the clarity of microscope imaging based on threshold selection according to claim 6, characterized in that: In step 4.1), the samples with stable microscopic structures but unevenly distributed within the microscope field of view include, for example, temporary slides of yeast solutions.

Images