Operation method for observing perforated plate through microscope

Abstract

The invention relates to the technical field of microscope imaging, and discloses an operation method for observing a perforated plate by a microscope, which comprises the following steps: dripping a culture into pores of the perforated plate, and putting the perforated plate with the culture on an electric control objective table of the microscope; automatically scanning holes by using a microscope low-power lens, and scanning each hole to obtain a first image; through automatic processing of the first image, whether a culture exists in each hole of the perforated plate or not is analyzed, and the analysis result comprises the position of the center of the culture in the hole containing the culture; selecting a hole needing to be shot according to an analysis result; and shooting the hole needing to be shot by using a microscope high-power lens to obtain a second image, and processing and analyzing the second image to obtain an experimental conclusion. According to the method, naked eye identification errors are reduced, the overall process steps are simplified, and a large amount of time is also saved.

Description

Technical Field

[0001] The present application relates to the technical field of microscope imaging, and more specifically to an operating method for observing a multi-well plate under a microscope. Background Art

[0002] Since the 21st century, chip technology has developed rapidly, and computer computing and cloud storage capabilities have also been significantly improved, giving rise to deep learning. In just twenty years, the information revolution has transformed into an intelligent revolution. As the latest subcategory of artificial intelligence (AI), deep learning algorithms do not require human-defined features and are driven by structured big data. They have great potential in pattern recognition and representation learning. Deep learning has shown great advantages in interpreting medical imaging data, assisting in optimizing diagnosis and treatment processes, and reducing medical errors.

[0003] In the field of image processing (CV), deep learning plays an important role in image classification and localization, object detection, object tracking, semantic segmentation, and instance segmentation, such as the automatic segmentation and image registration used in this technology.

[0004] In recent years, with the rapid development of electronic control technology, computer technology and image digitization technology, the demand for more convenient microscope operation procedures has increased. Convenient and fast microscope operation procedures can quickly improve the research efficiency of experimenters and medical workers. Summary of the invention

[0005] The purpose of the present application is to provide an operating method for observing a multi-well plate under a microscope, which reduces errors caused by direct observation by human eyes, simplifies the overall process steps, and saves a lot of time.

[0006] The present application discloses an operating method for observing a multi-well plate under a microscope, comprising:

[0007] dripping the culture into the wells of a multi-well plate, and placing the multi-well plate with the culture on an electronically controlled stage of a microscope;

[0008] Use a low-power microscope to automatically scan the wells, scan each well, and obtain a first image;

[0009] Analyzing whether the culture is present in each well of the multi-well plate by automatically processing the first image, wherein the analysis result includes the position of the center of the culture in the well containing the culture;

[0010] Select the hole to be photographed according to the analysis result;

[0011] The hole to be photographed is photographed using a high-power microscope to obtain a second image, and the second image is processed and analyzed to obtain an experimental conclusion.

[0012] In a preferred example, after automatically scanning the wells using a low-power microscope, the method further includes: inputting basic parameters of the multi-well plate into a microscope operating program, wherein the basic parameters include the style and specifications of the multi-well plate.

[0013] In a preferred embodiment, photographing the hole to be photographed using a microscope with a high power lens to obtain a second image and processing and analyzing the second image further includes:

[0014] Position the focus of the high-power microscope directly on the center of the culture in the well to be photographed;

[0015] According to the different imaging effects of the culture in different wells, the parameters brightness and focus were adjusted to be balanced parameters;

[0016] Calculate a parameter average value according to the parameters set in the different holes;

[0017] The parameters are set to the parameter average value, and the camera imaging under the high-power microscope is performed on each selected hole to obtain a second image;

[0018] By importing the second image, fusing all the selected second images to obtain a three-dimensional image, and performing automatic segmentation and annotation;

[0019] After the segmentation and annotation of the 3D images of the same well taken at different times are completed, the annotated information is calculated for comparison, and multiple 3D images are superimposed and compared;

[0020] Based on the labeled comparison information, the state changes of the organoids are analyzed and the conclusions of the experiment are obtained.

[0021] In a preferred embodiment, the automatic processing of the first image to analyze whether the culture is present in each well of the multi-well plate further comprises:

[0022] determining whether the culture is present in each well of the multi-well plate by judging whether the image of the well is clear by an edge detection method;

[0023] The first image is processed based on the edge detection method to identify all pixel points at the edge of the culture in the first image, and then the average value of the horizontal coordinate and the average value of the vertical coordinate of the pixel points at the edge are calculated, and the two average values ​​are recorded as the position of the center of the culture.

[0024] In a preferred embodiment, the step of importing the second image, fusing all selected second images to obtain a three-dimensional image, and automatically segmenting and labeling the image further comprises:

[0025] First, all the second images at different heights taken in each hole are fused, that is, in a manner simulating CT images, the interval of taking the second images is set to the thickness of each image, and a three-dimensional stereo image is obtained after three-dimensional reconstruction;

[0026] The three-dimensional image is annotated and segmented to obtain three-dimensional stereo annotations.

[0027] In a preferred embodiment, after the segmentation and annotation of the three-dimensional images of the same hole taken at different times is completed, the annotated information is calculated for comparison, and multiple three-dimensional images are superimposed and compared, further comprising:

[0028] Processing the three-dimensional stereoscopic image, and comparing each pixel point of each annotation in the three-dimensional stereoscopic image taken at different times;

[0029] Based on the calculated similarity, determine whether it is the same annotation and unify the same annotation ID;

[0030] The three-dimensional images with the same labeled ID are fused through an image fusion function, each of the three-dimensional images is given a different weight, and multiple three-dimensional images are superimposed and merged to display the merged image.

[0031] In a preferred example, before the second image is fused by the image fusion function, the method further includes: generating images of the same hole in different color channels.

[0032] In a preferred example, the annotation and segmentation method is a 3D UXNet algorithm.

[0033] In a preferred example, the comparison algorithm is the DeepSort algorithm.

[0034] In a preferred example, the image fusion function is provided by OpenCV.

[0035] Compared with the prior art, the main differences and effects of the embodiments of the present application are:

[0036] This method reduces the errors of naked eye identification and improves the accuracy of analysis through image processing;

[0037] Furthermore, the integrated operation process solves the problems of long operation time and lack of convenience in current microscopic measurement, improves the efficiency of microscope users, simplifies the overall process steps, and saves a lot of time.

[0038] Furthermore, the standardization of operations is improved.

[0039] The specification of the present application records a large number of technical features, which are distributed in various technical solutions. If all possible combinations of technical features of the present application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of the present application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions should all be regarded as having been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be regarded as having been recorded because it is technically infeasible, and the solution of A+B+C+E should be regarded as having been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a schematic flow chart of an operation method for observing a multi-well plate under a microscope according to an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of the effect of an edge extraction method according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of a process of capturing, obtaining, processing and analyzing a second image according to an embodiment of the present application;

[0044] Figure 4 is a schematic diagram of the principle of automatic segmentation in one embodiment of the present application;

[0045] Figure 5 is a schematic diagram comparing the effects of automatic segmentation in one embodiment of the present application;

[0046] Figure 6 is a schematic diagram of a process of superimposing and comparing three-dimensional images according to an embodiment of the present application;

[0047] FIG. 7 is a schematic diagram of the principle of image annotation and registration according to an embodiment of the present application; wherein FIG. 7( a ) is a schematic diagram showing a comparison of images of the same template hole taken at different times, and FIG. 7( b ) shows parameters related to annotation of the two images;

[0048] Figure 8 It is a schematic diagram of the principle of image superposition according to an embodiment of the present application.

[0049] In the accompanying drawings, the symbols are as follows:

[0050] 1~5- Operation steps of microscopic observation of multi-well plates;

[0051] 501-507-process steps of capturing a second image and processing and analyzing the image;

[0052] 506.1 to 506.3 - Process steps for superimposing and comparing three-dimensional images; DETAILED DESCRIPTION

[0053] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0055] The present application discloses a method for operating a multi-well plate by microscope observation, such as Figure 1 As shown, including:

[0056] Step 1: drop the culture into the wells of a multi-well plate, and place the multi-well plate with the culture on the electronically controlled stage of a microscope.

[0057] Step 2: Use a low-power microscope to automatically scan the wells, scan each well, and obtain a first image.

[0058] In one embodiment, after using a low-power microscope to automatically scan the wells, the method further includes: inputting basic parameters of the multi-well plate into a microscope operation program, the basic parameters including the style and specifications of the multi-well plate.

[0059] Step 3, by automatically processing the first image, analyzing whether there is a culture in each well of the multi-well plate, and the analysis results include the position of the center of the culture in the well containing the culture.

[0060] In one embodiment, the step of automatically processing the first image to analyze whether there is a culture in each well of the multi-well plate further comprises:

[0061] Whether there is culture in each well of the multi-well plate is determined by edge detection method to determine whether the image of the well is clear.

[0062] The first image is processed based on the edge detection method to identify all pixel points at the edge of the culture in the first image, and then the average values ​​of the horizontal coordinates and the average values ​​of the vertical coordinates of the edge pixels are calculated, and the two average values ​​are recorded as the position of the center of the culture.

[0063] The effect of edge detection method can be referred to Figure 2 .

[0064] Step 4: Select the holes to be photographed based on the analysis results.

[0065] Step 5, use a high-power microscope to photograph the holes to be photographed to obtain a second image, process and analyze the second image to obtain an experimental conclusion.

[0066] In one embodiment, the images are processed using a program tool based on the Qt framework and Python that automatically processes subsequent images using a variety of algorithms.

[0067] In one embodiment, the hole to be photographed is photographed using a high-power microscope to obtain a second image and the steps of processing and analyzing the second image are as follows: Figure 3 As shown, further comprising:

[0068] Step 501, directly position the focus of the high-power microscope to the center of the culture in the well to be photographed.

[0069] Step 502, according to the different imaging effects of the cultures in different wells, adjust the parameters of brightness and focus to set them as balanced parameters.

[0070] Step 503, obtaining a parameter average value according to the parameters set in different holes.

[0071] Step 504 , setting the parameters to the parameter average value, performing camera imaging under a high-power microscope on each selected hole to obtain a second image.

[0072] Step 505 , by importing the second images, fusing all the selected second images to obtain a three-dimensional image, and performing automatic segmentation and annotation.

[0073] In one embodiment, the step of importing the second image, fusing all the selected second images to obtain a three-dimensional image, and automatically segmenting and labeling the image further includes:

[0074] like Figure 4 As shown, all the second images at different heights taken in each hole are first fused, that is, in a manner simulating CT images, the interval between the second image shots is set to the thickness of each image, and a three-dimensional stereoscopic image is obtained after three-dimensional reconstruction. Figure 4 The upper left is the x,y plane, the lower left is the x,z plane, the lower right is the y,z plane, and the upper right is the 3D reconstructed effect, that is, the x,y plane is a single layer scanned by a microscope, and then the z axis is scanned multiple times at a certain distance and finally superimposed together.

[0075] The three-dimensional image is labeled and segmented to obtain three-dimensional stereo annotations.

[0076] In one embodiment, the method for labeling and segmenting is a 3D UXNet algorithm. 3D UXNet is a lightweight 3D convolutional neural network that uses a ConvNet module to adjust a hierarchical Transformer to achieve robust voxel segmentation.

[0077] like Figure 5 As shown, the left picture is the organoid image exported after microscope photography, and the right picture is the image after automatic segmentation by the program

[0078] The image on the left is the original image. It is relatively difficult to obtain information intuitively. Researchers will segment and annotate the image before obtaining information.

[0079] Through the automatic segmentation program, objects in the image can be quickly marked. Compared with traditional manual marking, the efficiency of image marking has been greatly improved. The analysis of the algorithm also improves the reliability of the marking results and avoids marking errors caused by naked eye recognition.

[0080] Step 506 , after the segmentation and annotation of the three-dimensional images of the same hole taken at different times are completed, the annotated information is calculated for comparison, and the multiple three-dimensional images are superimposed and compared.

[0081] In one embodiment, after the segmentation and annotation of the three-dimensional images of the same hole taken at different times is completed, the annotated information is calculated for comparison, and multiple three-dimensional images are superimposed and compared, such as Figure 6 As shown, further comprising:

[0082] Step 506.1, processing the three-dimensional stereo image, and comparing each pixel point of each annotation in the three-dimensional stereo images taken at different times.

[0083] In one embodiment, the algorithm used for comparison is the DeepSort algorithm.

[0084] Step 506.2: determine whether they are the same annotation based on the calculated similarity, and unify the same annotation IDs.

[0085] Figure 7(a) shows images of the same template hole taken at different times, such as the images taken on the first day and the Nth day. Figure 7(b) shows the parameters related to the annotations of the two images, which is convenient for experimenters to compare smaller annotations, such as matching based on position, and generally agreeing that they are the same organoid. The same organoid can be compared and the changes in information such as the major axis and minor axis can be calculated. Based on these data changes, the changes in the organoid can be judged, making it convenient for observers to view key organoids.

[0086] Step 506.3, perform fusion processing on the three-dimensional stereo images with the same labeled id through the image fusion function, give different weights to each three-dimensional stereo image, superimpose and merge multiple three-dimensional stereo images, and display the merged image.

[0087] Figure 8 The image obtained by superimposing images under different channels, with different colors superimposed together, is more convenient for the experimenter to make a comparison through colors and judge the state of the organoid. In one embodiment, before fusing the second image through the image fusion function, it also includes: generating images under different color channels of the same hole.

[0088] In one embodiment, the image fusion function is provided by OpenCV.

[0089] Step 507, analyze the labeled comparison information to obtain the state change of the organoid and the conclusion of the experiment.

[0090] It should be noted that in the claims and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0091] All documents mentioned in this application are cited as references in this application, just as each document is cited as reference individually. In addition, it should be understood that after reading the above teaching content of this application, those skilled in the art can make various changes or modifications to this application, and these equivalent forms also fall within the scope defined by the claims attached to this application.

[0092] In some cases, the actions or steps described in the claims may be performed in a different order than in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for operating a multi-well plate under a microscope, characterized in that: include: dripping the culture into the wells of a multi-well plate, and placing the multi-well plate with the culture on an electronically controlled stage of a microscope; Use a low-power microscope to automatically scan the wells, scan each well, and obtain a first image; Analyzing whether the culture is present in each well of the multi-well plate by automatically processing the first image, wherein the analysis result includes the position of the center of the culture in the well containing the culture; Select the hole to be photographed according to the analysis result; The hole to be photographed is photographed using a high-power microscope to obtain a second image, and the second image is processed and analyzed to obtain an experimental conclusion.

2. The operating method according to claim 1, characterized in that: After using the low-power microscope to automatically scan the wells, the process further includes: inputting basic parameters of the multi-well plate into a microscope operation program, wherein the basic parameters include the style and specifications of the multi-well plate.

3. The operating method according to claim 1, characterized in that: The method of photographing the hole to be photographed using a microscope with a high power lens to obtain a second image and processing and analyzing the second image further includes: Position the focus of the high-power microscope directly on the center of the culture in the well to be photographed; According to the different imaging effects of the culture in different wells, the parameters brightness and focus were adjusted to be balanced parameters; Calculate a parameter average value according to the parameters set in the different holes; The parameters are set to the parameter average value, and the camera imaging under the high-power microscope is performed on each selected hole to obtain a second image; By importing the second image, fusing all the selected second images to obtain a three-dimensional image, and performing automatic segmentation and annotation; After the segmentation and annotation of the 3D images of the same well taken at different times are completed, the annotated information is calculated for comparison, and multiple 3D images are superimposed and compared; Based on the labeled comparison information, the state changes of the organoids are analyzed and the conclusions of the experiment are obtained.

4. The operating method according to claim 1, characterized in that: The automatic processing of the first image to analyze whether the culture is present in each well of the multi-well plate further comprises: determining whether the culture is present in each well of the multi-well plate by judging whether the image of the well is clear by an edge detection method; The first image is processed based on the edge detection method to identify all pixel points at the edge of the culture in the first image, and then the average value of the horizontal coordinate and the average value of the vertical coordinate of the pixel points at the edge are calculated, and the two average values ​​are recorded as the position of the center of the culture.

5. The operating method according to claim 3, characterized in that: The method further includes: importing the second image, fusing all the selected second images to obtain a three-dimensional image, and automatically segmenting and labeling the image; First, all the second images at different heights taken in each hole are fused, that is, in a manner simulating CT images, the interval of taking the second images is set to the thickness of each image, and a three-dimensional stereo image is obtained after three-dimensional reconstruction; The three-dimensional image is annotated and segmented to obtain three-dimensional stereo annotations.

6. The operating method according to claim 3, characterized in that: After the segmentation and annotation of the three-dimensional images of the same hole taken at different times is completed, the annotated information is calculated for comparison, and multiple three-dimensional images are superimposed and compared, further comprising: Processing the three-dimensional stereoscopic image, and comparing each pixel point of each annotation in the three-dimensional stereoscopic image taken at different times; Based on the calculated similarity, determine whether it is the same annotation and unify the same annotation ID; The three-dimensional images with the same labeled ID are fused through an image fusion function, each of the three-dimensional images is given a different weight, and multiple three-dimensional images are superimposed and merged to display the merged image.

7. The operating method according to claim 6, characterized in that: Before the second image is fused by the image fusion function, the method further includes: generating images of the same hole under different color channels.

8. The operating method according to claim 5, characterized in that: The annotation segmentation method is the 3D UXNet algorithm.

9. The operating method according to claim 6, characterized in that: The algorithm used for comparison is the DeepSort algorithm.

10. The operating method according to claim 5 or 6, characterized in that: The image fusion function is provided by OpenCV.

Images