Depth-of-field fusion method for electron microscope image
The detection frame of the organoid is obtained through example segmentation and local depth of field fusion is solved, and the spot and focus problems of existing electron microscopes when observing organoids are improved, the clarity of the image is improved, making organoids more convenient for observation.
Patent Information
- Application Number
- CN202311011395.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-11
- Publication Date
- 2025-02-18
AI Technical Summary
When observing organoids with existing electron microscopes, the spot problem and focus problem caused by the layer scanning method affect the clarity of the depth of field fusion picture, and it is impossible to identify and distinguish the clear and fuzzy parts of the sweep layer.
Obtain the detection box of a single organoid through instance segmentation, find out whether there are organoids in the same position of the other layers according to the location of the detection box, and local depth of field fusion of all layers containing organoids, eliminate the layers that do not exist, and only the organoids are in the organoids. The area that appears undergoes local depth of field fusion.
It improves the clarity of electron microscope images, makes organoids more convenient to observe, and solves the spot and focus problems caused by laminar scanning.
Smart Images
Figure CN119477712A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of electron microscope imaging, and more specifically to a depth of field fusion method for electron microscope images. Background Art
[0002] As the latest subcategory of artificial intelligence (AI), deep learning algorithms do not require human-defined features and are driven by structured big data. They have great potential in pattern recognition and representation learning. Deep learning has shown great advantages in interpreting medical imaging data, assisting in optimizing diagnosis and treatment processes, and reducing medical errors.
[0003] In the field of image processing (CV), deep learning plays an important role in image classification and localization, object detection, object tracking, semantic segmentation, and instance segmentation, such as the object detection and instance segmentation used in this technology.
[0004] Organoids are three-dimensional (3D) cell cultures that contain some key features of the organs they represent. Organoids are usually cultured in 96-well culture plates and need to be observed using an electron microscope. Existing electron microscopes use a layer scanning method to observe organoids, scanning the organoids into several layers of images with a fixed step length between each layer. This method has certain disadvantages, which is caused by the structure of the microscope. The imaging of the microscope will place two light sources above and below the culture dish, which will cause a large light spot to appear in the layer close to the light source in the layer scanned image, which will affect the clarity of the depth of field fused image to a certain extent. Due to focusing problems, the layer scan will inevitably have blurred parts. Existing electron microscopes cannot identify and distinguish the clear and blurred parts of the scanned layers, and will fuse all the scanned layers together in depth of field, which may cause the final depth of field fused photos to be out of focus, which is not conducive to observation. Summary of the invention
[0005] The purpose of this application is to provide a depth of field fusion method for electron microscope images, use instance segmentation to obtain a detection frame of a single organoid, find out whether there is an organoid at the same position in other layers based on the position of the detection frame, and perform local depth of field fusion on all layers containing organoids to make the fused photos clearer and easier to observe.
[0006] The present application discloses a depth of field fusion method for electron microscope images, comprising:
[0007] Culture organoids were dripped into wells of a multi-well plate;
[0008] Using an electron microscope to photograph the multi-well plate in layers to obtain all layer scan images;
[0009] Obtain a detection frame of a single organoid in a single-layer image by instance segmentation, and search for the organoid at the same position in other layers according to the position of the detection frame to determine the layer containing the organoid;
[0010] Eliminate the layer where the organoid does not exist, and perform local depth of field fusion on the images of all layers containing the organoid in the area where the organoid appears;
[0011] The area where local depth of field fusion has been performed is retained, and depth of field fusion is performed on other areas using all the layer scan images to obtain a clear panoramic image.
[0012] In a preferred embodiment, after dripping the cultured organoids into the wells of a multi-well plate, the method further comprises: placing the multi-well plate with the organoids on an electronically controlled stage of a microscope.
[0013] In a preferred embodiment, the step of obtaining a detection frame of a single organoid in a single-layer image by instance segmentation, and searching for the organoid at the same position in other layers according to the position of the detection frame to determine the layer containing the organoid, further comprises:
[0014] Perform target detection on each single-layer image, perform instance segmentation to obtain the detection frame, identify the organoid area, and mark it with different colors;
[0015] According to the position of the detection frame, find out whether there is a detection frame at the same position in other layers, and calculate the similarity;
[0016] Based on the calculated similarity, it is determined whether the label is the same organoid, and the same label ID is unified to determine the number of layers containing the organoid.
[0017] In a preferred embodiment, the step of removing the layer where the organoid does not exist and performing local depth of field fusion on all images of the layer containing the organoid in the area where the organoid appears further comprises:
[0018] Assume that the number of scanning layers of the microscope is Z, and the length and width of the scan are X and Y respectively. Determine the processing point P x,y Whether it is within the detection frame on the Z layer, if so, perform the following local depth of field fusion processing;
[0019] Superimpose Z images. If the number of layers containing the same organoid labeling id on the Z layer is continuous, select in represents the fusion function used, Z s is the highest layer of the organoid identified, Z e The lowest layer of the organoid identified;
[0020] If the number of layers containing the same organoid label id on the Z layer is discontinuous, then the P layer containing the organoid is selected. x,y The values of are merged;
[0021] If there are multiple organoids at the same point on the Z layer, the multiple organoids are fused separately and then superimposed, and the layer where the organoids do not exist is eliminated.
[0022] In a preferred embodiment, the method of retaining the area where local depth of field fusion has been performed and performing depth of field fusion on other areas using all the layer scan images to obtain a clear panoramic image further includes:
[0023] Assume that the number of microscope scanning layers is Z, and the length and width of the scan are X and Y respectively. Now perform the following operations:
[0024] Start processing point by point from (0,0) and determine the processing point P x,y Whether the organoid is identified on the Z layer;
[0025] For example, the processed point P x,y When the organoid is identified on the Z layer, local depth of field fusion processing is performed;
[0026] For example, the processed point P x,y If the organoid is not identified on the Z layer, the P value of the generated panoramic image is x,y The position is the fusion of Z points;
[0027] After processing all the points in the image, a clear panoramic image is obtained.
[0028] In a preferred example, the depth of field fusion method is an average method, a maximum method or a sobel operator method.
[0029] In a preferred example, the target detection method includes a regional convolutional network.
[0030] In a preferred embodiment, the porous plate is photographed in layers from top to bottom.
[0031] In a preferred embodiment, there is a fixed step length between each layer of the layer scan image.
[0032] In a preferred embodiment, the instance segmentation includes a combination of object detection and semantic segmentation. Compared with the prior art, the main differences and effects of the embodiments of the present application are:
[0033] Based on the traditional electron microscope, the present invention obtains the detection frame of a single organoid through instance segmentation, and searches for organoids at the same position in other layers according to the position of the detection frame. All layers with organoids are locally fused with depth of field to obtain a clearer picture, making the organoids easier to observe.
[0034] The specification of the present application records a large number of technical features, which are distributed in various technical solutions. If all possible combinations of technical features of the present application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of the present application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions should all be regarded as having been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be regarded as having been recorded because it is technically infeasible, and the solution of A+B+C+E should be regarded as having been recorded. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0036] Figure 1 is a schematic flow chart of a method for depth of field fusion of electron microscope images according to one embodiment of the present application;
[0037] Figure 2 is a schematic diagram of an organoid layer scan image according to one embodiment of the present application, wherein Figure 2 (a) to Figure 2 (f) Layer scan images of the 1st, 6th, 10th, 11th, 15th, and 20th layers, respectively;
[0038] Figure 3 is a schematic diagram of a process of performing local depth of field fusion on a layer containing organoids according to one embodiment of the present application;
[0039] Figure 4 is a schematic diagram of a process for forming a panoramic depth-of-field fusion image according to an embodiment of the present application;
[0040] Figure 5 is a schematic diagram of a layer scan image after fusion processing according to an embodiment of the present application, wherein Figure 5 (a) to Figure 5 (c) are the images after fusion of all the electron microscope layer scan depth of field, the image after instance segmentation, and the image after instance segmentation and fusion.
[0041] In the accompanying drawings, the symbols are as follows:
[0042] 1~5-Process steps of electron microscope image depth of field fusion method;
[0043] 401~407-process steps of local depth of field fusion;
[0044] 501-503-process of forming a panoramic depth-of-field fusion image; DETAILED DESCRIPTION
[0045] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.
[0046] Description of some concepts:
[0047] Object Detection: The basic task of object detection is to identify the category of the object being detected in the image, and to use a rectangular bounding box to determine the location and size of the object and give the corresponding confidence. Before deep learning methods were applied to the field of object detection, the field of object detection developed slowly. In the ImageNet classification task in 2012, the application of convolutional neural networks greatly improved the effect of image classification tasks. Driven by this, Girshick et al. used regional convolutional networks (regions with CNN features, R-CNN) in the field of object detection for the first time, achieving a huge improvement in detection results. Since then, the combination of deep learning and object detection tasks has led to the rapid development of the field of object detection and has been widely used in practice. Object detection has broad application prospects and has achieved good results in monitoring security, autonomous driving, remote sensing detection, medical image lesion detection and other fields.
[0048] Instance Segmentation: Image segmentation refers to dividing an image into several non-overlapping regions based on features such as grayscale, color, spatial texture, and geometric shape, so that these features show consistency or similarity in the same region, but show obvious differences between different regions. Object detection is to identify the content in the image and detect its location; semantic segmentation is to classify each pixel in the image by labeling it with a category; instance segmentation is a combination of object detection and semantic segmentation, detecting the object in the image (object detection) and then labeling each pixel (semantic segmentation). The purpose of instance segmentation is to detect the object in the input image and assign a category label to each pixel of the object. Instance segmentation can distinguish different instances with the same foreground semantic category, which is the biggest difference between it and semantic segmentation. Compared with semantic segmentation, instance segmentation developed later, so the instance segmentation model is mainly based on deep learning technology, but it is also an important component of image segmentation.
[0049] Depth of field fusion: Depth of field fusion is to analyze the image sequence of non-planar objects collected when the microscope lens is continuously zoomed, extract the relatively clear focused areas in each frame of the sequence, and then compete for focus clarity and image fusion for these areas according to their positions to form a new full-depth image with clear areas.
[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0051] The present application discloses a method for depth of field fusion of electron microscope images, such as Figure 1 As shown, the steps include:
[0052] In step 1, culture organoids are dripped into the wells of a multi-well plate.
[0053] In one embodiment, after dropping the cultured organoids into the wells of the multi-well plate, the method further comprises: placing the multi-well plate with the organoids on an electronically controlled stage of a microscope.
[0054] Step 2: Use an electron microscope to photograph the porous plate in layers to obtain all layer scan images.
[0055] In one embodiment, the multi-well plate is photographed in layers from top to bottom. In one embodiment, there is a fixed step length between each layer of the layer scan image.
[0056] In one embodiment, Figure 2 As shown, from Figure 2 (a) to Figure 2 (f) are the layer scan images of the 1st, 6th, 10th, 11th, 15th, and 20th layers respectively; it can be seen from the figure that Figure 2 (a) There are no clear individual organoids. Figure 2 (b) to Figure 2 (e) Clear individual organoids gradually increase to Figure 2 The clear individual organoids in (f) gradually decrease.
[0057] Step 3: Obtain the detection frame of a single organoid in a single-layer image through instance segmentation, and find out whether there is an organoid at the same position in other layers based on the position of the detection frame to determine the layer containing the organoid.
[0058] In one embodiment, the step of obtaining a detection frame of a single organoid in a single-layer image by instance segmentation, and searching for organoids at the same position of other layers according to the position of the detection frame to determine the layer containing the organoid further comprises:
[0059] Step 301 , perform target detection on each single-layer image, perform instance segmentation to obtain a detection frame, identify the organoid area, and mark it with different colors.
[0060] In one embodiment, a method for object detection includes a region convolutional network.
[0061] In one embodiment, instance segmentation includes a combination of object detection and semantic segmentation.
[0062] Step 302, based on the position of the detection frame, find out whether there is a detection frame at the same position in other layers, and calculate the similarity.
[0063] Step 303, based on the calculated similarity, determine whether they are annotations of the same organoid, unify the same annotation IDs, and determine the number of layers containing organoids.
[0064] Step 4: Eliminate the layers without organoids and perform local depth-of-field fusion on the images of all layers containing organoids in the areas where the organoids appear.
[0065] In one embodiment, the layer without organoids is removed, and the images of all layers containing organoids are locally fused in the area where the organoids appear, such as Figure 3 As shown, further comprising:
[0066] Assume that the number of microscope scanning layers is Z, the length and width of the scan are X and Y respectively, and the processing starts from (0,0) and is performed point by point.
[0067] Step 401, determine the processing point P x,y Whether it is within the detection frame on the Z layer, if it is, the following local depth of field fusion processing is performed.
[0068] Step 402, remove the layer without organoids and superimpose the remaining M images.
[0069] Step 403, determining whether the organoid labeling ids contained in the M layer are the same.
[0070] Step 404: If they are the same, determine whether the number of layers containing organoids on layer M is continuous.
[0071] Step 405, if continuous, select in represents the fusion function used, Z s Z is the highest layer of the organoid identified. e is the lowest layer of the organoid identified.
[0072] In one embodiment, the depth of field fusion method is an averaging method, a maximum value method, a Sobel operator method or other methods.
[0073] Step 406: If it is not continuous, select the layer P containing the organoid x,y The values are merged.
[0074] Step 407: If they are not the same, it means that there are multiple organoids at the same point on the M layer. It is determined whether the multiple organoids are continuous, and then they are fused separately and then superimposed.
[0075] Step 5: retain the area where local depth of field fusion has been performed, and use all layer scan images to perform depth of field fusion on other areas to obtain a clear panoramic image.
[0076] In one embodiment, the area where local depth of field fusion has been performed is retained, and all layer scan images are used to perform depth of field fusion on other areas to obtain a clear panoramic image, such as Figure 4 As shown, further comprising:
[0077] Assume that the number of microscope scanning layers is Z, and the length and width of the scan are X and Y respectively. Now perform the following operations:
[0078] Starting from (0,0), we process point by point.
[0079] Step 501, determine the processing point P x,y Whether organoids are identified in the Z layer;
[0080] Step 502, as the processed point P x,y If organoids are identified on the Z layer, local depth of field fusion processing is performed;
[0081] Step 503, as the processed point P x,y If no organoids are identified on the Z layer, the P value of the generated panoramic image is x,y The position is the fusion of Z points;
[0082] After processing all the points in the image, a clear panoramic image is obtained.
[0083] Figure 5 The pictures show the process using different processing steps, from Figure 5 (a) to Figure 5 (c) are the images after the depth of field fusion of all electron microscope layer scans, the image after instance segmentation, and the fusion image after instance segmentation. Obviously, the image after depth of field fusion is clearer and easier to observe than the original image.
[0084] It should be noted that in the claims and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0085] All documents mentioned in the present invention are cited as references in this application, just as each document is cited as reference individually. In addition, it should be understood that after reading the above teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the claims attached to this application.
Claims
1. A depth of field fusion method for electron microscope images, characterized in that: include: Culture organoids are dripped into wells of a multi-well plate; Using an electron microscope to photograph the multi-well plate in layers to obtain all layer scan images; Obtain a detection frame of a single organoid in a single-layer image by instance segmentation, and search for the organoid at the same position in other layers according to the position of the detection frame to determine the layer containing the organoid; Eliminate the layer where the organoid does not exist, and perform local depth of field fusion on the images of all layers containing the organoid in the area where the organoid appears; The area where local depth of field fusion has been performed is retained, and depth of field fusion is performed on other areas using all the layer scan images to obtain a clear panoramic image.
2. The depth of field fusion method according to claim 1, characterized in that: After dropping the cultured organoids into the wells of the multi-well plate, the method further includes: placing the multi-well plate with the organoids on an electronically controlled stage of a microscope.
3. The depth of field fusion method according to claim 1, characterized in that: The method further comprises: obtaining a detection frame of a single organoid in a single-layer image by instance segmentation, and searching for the organoid at the same position of other layers according to the position of the detection frame to determine the layer containing the organoid. Perform target detection on each single-layer image, perform instance segmentation to obtain the detection frame, identify the organoid area, and mark it with different colors; According to the position of the detection frame, find out whether there is a detection frame at the same position in other layers, and calculate the similarity; Based on the calculated similarity, it is determined whether the label is the same organoid, and the same label ID is unified to determine the number of layers containing the organoid.
4. The depth of field fusion method according to claim 1, characterized in that: The step of eliminating the layer where the organoid does not exist and performing local depth of field fusion on all images of the layer containing the organoid in the area where the organoid appears further comprises: Assume that the number of scanning layers of the microscope is Z, and the length and width of the scan are X and Y respectively. Determine the processing point P x,y Whether it is within the detection frame on the Z layer, if so, perform the following local depth of field fusion processing; Superimpose Z images. If the number of layers containing the same organoid labeling id on the Z layer is continuous, select in represents the fusion function used, Z s is the highest layer of the organoid identified, Z e The lowest layer of the organoid identified; If the number of layers containing the same organoid label id on the Z layer is discontinuous, then the P layer containing the organoid is selected. x,y The values of are merged; If there are multiple organoids at the same point on the Z layer, the multiple organoids are fused separately and then superimposed, and the layer where the organoids do not exist is eliminated.
5. The depth of field fusion method according to claim 1, characterized in that: The method further includes: retaining the area where local depth of field fusion has been performed, and performing depth of field fusion on other areas using all the layer scan images to obtain a clear panoramic image. Assume that the number of microscope scanning layers is Z, and the length and width of the scan are X and Y respectively. Now perform the following operations: Start processing point by point from (0,0) and determine the processing point P x,y Whether the organoid is identified on the Z layer; For example, the processed point P x,y When the organoid is identified on the Z layer, local depth of field fusion processing is performed; For example, the processed point P x,y If the organoid is not identified on the Z layer, the P value of the generated panoramic image is x,y The position is the fusion of Z points; After processing all the points in the image, a clear panoramic image is obtained.
6. The depth of field fusion method according to claim 1, characterized in that: The depth of field fusion method is an average method, a maximum method or a sobel operator method.
7. The depth of field fusion method according to claim 3, characterized in that: The target detection method includes a regional convolutional network.
8. The depth of field fusion method according to claim 1, characterized in that: The multi-well plate was photographed layer by layer in order from top to bottom.
9. The depth of field fusion method according to claim 1, characterized in that: There is a fixed step length between each layer of the layer scan image.
10. The depth of field fusion method according to claim 1, characterized in that: The instance segmentation includes the combination of object detection and semantic segmentation.