Object tracking using image segmentation

Abstract

Object tracking using image segmentation is disclosed. A captured image of a specimen is obtained. A segmented image is generated based on the captured image. The segmented image indicates segments corresponding to objects of interest. One or more target objects are identified from the objects of interest in the segmented image. An object of interest can be identified that has a position and / or shape most similar to a target object shown in a previous image. Alternatively, an object of interest can be identified that is associated with a connection vector most similar to a connection vector connecting the target object in the previous image. A movement vector is drawn from a target object position in the previous image to a target object position in the segmented image. A field of view of a microscope is moved relative to the specimen in accordance with the movement vector to capture another image of the specimen.

Description

Technical Field

[0001] This disclosure relates to sample image acquisition. More specifically, this disclosure relates to object tracking using image segmentation. Background Technology

[0002] Microscopy is the field of science that uses microscopes to better observe objects that are difficult to see with the naked eye. Different branches of microscopy include, for example, optical microscopy, charged particle (electron and / or ion) microscopy, and scanning probe microscopy.

[0003] Charged particle microscopy involves using an accelerated beam of charged particles as an illumination source. Types of electron microscopes include, for example, transmission electron microscopy, scanning electron microscopy, scanning transmission electron microscopy, and focused ion beam microscopy.

[0004] The components of a transmission electron microscope (TEM) include: an electron optics column, a vacuum system, necessary electronics (lens sources for focusing and deflecting the beam and a high-voltage generator for the electron source), and control software. One end of the electron optics column contains an electron gun, and the other end contains an observation device (such as a camera). The electron beam is emitted from the electron gun and passes through a thin sample to transport electrons that are collected, focused, and projected onto the observation device. The entire electron path from the gun to the camera is under vacuum.

[0005] Similar to TEM, the components of a scanning electron microscope (SEM) include: an electron optical column, a vacuum system, necessary electronics (lens sources for focusing and deflecting the beam and a high-voltage generator for the electron source), and control software. The electron gun is located at one end of the electron optical column. The sample is located at the other end of the electron column. The electron beam from the electron gun is focused into a tiny spot on the sample surface. The electron beam scans above the sample in a rectangular grating. The intensities of various signals generated by the interaction between the electron beam and the sample are measured and stored in computer memory. The stored values ​​are then mapped to changes in brightness on the image display.

[0006] Scanning transmission electron microscopy (STEM) is similar to TEM, where images are formed by passing electrons through a sufficiently thin sample. However, unlike TEM, STEM focuses an electron beam onto a tiny spot of light and then scans the electron beam above the sample using a grating illumination system.

[0007] Focused ion beam microscopy (FIB microscopy) is similar to SEM; however, FIB microscopy uses an ion beam instead of an electron beam. Examples of ion beam sources include liquid metal ion sources (LMIS), such as gallium ion sources.

[0008] Microscopes are associated with a variety of configurable microscope parameters. Examples of SEM microscope parameters include: accelerating voltage (the voltage at which electrons are accelerated as they pass through an electron optics column); electron beam convergence angle; beam current; spot size (the diameter of the beam spot on the sample); residence time; and resolution. Different values ​​for these microscope parameters result in images of varying quality and properties. For example, higher magnification requires a smaller spot size. Higher signal-to-noise ratio and contrast resolution require a larger beam current. However, reducing the spot size also reduces the beam current.

[0009] Various methods can be used to obtain a three-dimensional (3D) rendering of a specimen. As an example, block-face scanning electron microscopy involves mounting the specimen in the vacuum chamber of a microscope; capturing an image of a block facet of the specimen using the microscope; cutting a thin slice from the block facet; raising the specimen so that a new block facet returns to the focal plane of the microscope; and capturing another image of the new block facet. The process is repeated until the entire 3D volume has been captured. As another example, sequential cross-sectional scanning electron microscopy involves slicing the specimen into thin slices; mounting the first slice in the vacuum chamber of the microscope; capturing an image of the surface of the first slice; mounting the next slice in the vacuum chamber of the microscope; and capturing another image of the surface of the current slice. The process is repeated until images of all slices have been captured. The term "cross-sectional surface" as used herein refers to a portion of the specimen captured in each 2D image, for example, a block facet in block-face scanning electron microscopy, or the surface of a slice in sequential cross-sectional scanning electron microscopy. The 2D images of the cross-sectional surfaces are stacked together to generate a 3D rendering of the specimen. 3D renderings can be presented at a user interface, printed on paper, and / or otherwise provided to users and / or another application. 3D renderings can be presented as 3D models and / or animations showing the cross-sectional surfaces of the specimen from top to bottom. 3D renderings can be interactive; for example, users can rotate the 3D model and / or rewind, stop, and fast-forward the animation.

[0010] Various methods can be used to obtain continuous images of a specimen changing over time. In-situ electron microscopy is a research technique that uses an electron microscope to observe the response of a specimen to a stimulus. The specimen is mounted within the microscope's field of view. A stimulus is activated, causing the specimen to respond. The microscope can capture images of the specimen at periodic time intervals, such as every 0.1 seconds. The captured images are collected together to form an animation. The animation shows the changes that occur within the specimen over time.

[0011] One of the drawbacks associated with 3D rendering of specimens is the limited field of view of the microscope. The target object (the object the user wants to observe) can appear in different locations across different cross-sectional surfaces of the specimen. While the target object may appear in the field of view of a particular cross-sectional surface, it may also be outside the field of view of another cross-sectional surface. Therefore, the target object is not fully captured within the specimen depth in the 3D rendering.

[0012] One of the drawbacks associated with obtaining continuous images of a specimen is the limited field of view of the microscope. The target object can move to different locations within the time period of interest. While the target object may appear in the field of view at a specific time interval, it may also be out of the field of view at another time interval. Therefore, the target object cannot be fully captured within the time period of interest.

[0013] Another drawback associated with obtaining 3D renderings of the specimen and acquiring continuous images of the specimen over time is the extreme difficulty in tracking the target object. Multiple objects of interest may appear on the cross-sectional surface. Other miscellaneous objects may also appear on the cross-sectional surface. Capturing specimen images may also involve noise. Therefore, it is difficult to determine which of the objects of interest shown in the specimen image is the target object. As an example, multiple mitochondria may appear on the cross-sectional surface of brain tissue. However, a user may be particularly interested in one of the mitochondria. Due to the similarity of mitochondria, it is difficult to distinguish the mitochondria of particular interest from the other mitochondria and other objects shown in the captured image, as well as from the noise. As another example, multiple bubbles may move within the specimen due to stimulation. However, a user may be particularly interested in a particular group of bubbles. Due to the similarity of the bubbles, it is difficult to identify the group of bubbles of particular interest to the user.

[0014] The objective of this invention is to improve the process of obtaining 3D renderings of samples. More specifically, the objective is to provide an improved method for obtaining 3D renderings that better track target objects.

[0015] The objective of this invention is to improve the process of obtaining continuous images of a specimen that change over time. More specifically, the objective is to provide an improved method for obtaining continuous images of a specimen that better tracks a target object.

[0016] To this end, the present invention provides a method according to technical solution 1. The method according to technical solution 1 includes the step of obtaining a first image corresponding to a specimen. A first position corresponding to a target object in the first image is identified. Then, a second image corresponding to the specimen (captured by a microscope) is obtained. Image segmentation techniques are applied to the second image to obtain a segmented image. The segmented image includes (a) a first set of segments corresponding to an object of interest and (b) a second set of segments not corresponding to any object of interest. A specific object of interest associated with the highest similarity score of the target object shown in the first image among the objects of interest shown in the segmented image is identified as the target object shown in the segmented image. A second position corresponding to the target object in the segmented image is identified. A movement vector from the first position in the first image to the second position in the segmented image is determined. The field of view of the microscope is moved relative to the specimen according to the movement vector to capture a third image corresponding to the specimen. In an embodiment, the method may be performed by at least one device including a hardware processor. Using the above method, it is possible to accurately identify and track target objects through different cross-sectional surfaces of the specimen. Furthermore, using the above method, it is possible to accurately identify and track target objects through specimens that change over time. Since the target object is not directly identified from the captured image, accuracy is increased; in fact, the target object is identified from the segmented image generated based on the captured image. Image segmentation is used to filter out uninteresting objects shown in the captured image. The segmented image therefore involves less "noise" in the process of identifying the target object. Based on this method, the target object remains within the microscope's field of view.

[0017] A first image corresponding to the specimen shows the first cross-sectional surface of the specimen. A second image corresponding to the specimen shows the second cross-sectional surface of the specimen. Thus, the method is applied to different cross-sectional surfaces of the specimen, and a 3D rendering of the specimen is obtained. The target object is tracked through the 3D rendering of the specimen. The method includes the following steps: cutting thin slices from the block surface of the specimen to remove the first cross-sectional surface of the specimen and expose the second cross-sectional surface of the specimen.

[0018] In this embodiment, after moving the microscope's field of view relative to the specimen according to the motion vector, a third image corresponding to the specimen is captured. The third image shows the second cross-sectional surface of the specimen. Thus, two images are captured from the second cross-sectional surface. The "second image" is captured for the purpose of identifying the target object and determining the motion vector. The "third image" is captured after centering the target object based on the motion vector. Therefore, an image accurately centered on the target object is captured.

[0019] In this embodiment, a third image corresponding to the specimen is obtained. The third image shows the second cross-sectional surface of the specimen. The third image is captured by the microscope after moving the microscope's field of view relative to the specimen according to the motion vector. A set of images for tracking the target object is compiled. The set of images includes the third image but does not include the second image. Thus, two images are captured from the second cross-sectional surface. The "second image" is captured for the purpose of identifying the target object and determining the motion vector. The "third image" is captured after centering the target object based on the motion vector. When compiling the set of images for tracking the target object, only images centered on the target object are included; images taken for the purpose of determining the motion vector (and not necessarily centered on the target object) are not included. Therefore, the set of images for tracking the target object only includes images centered on the target object.

[0020] In this embodiment, after moving the microscope's field of view relative to the specimen according to the movement vector, a third image corresponding to the specimen is captured. The third image shows the third cross-sectional surface of the specimen. Thus, only one image is captured from the second cross-sectional surface. Any deviation of the target object shown in the second image from the center is accepted. It is assumed that the positional change of the target object from one cross-sectional surface to the next is insignificant or sufficiently small to be easily corrected in the image post-processing step. Therefore, the field of view used to capture the third cross-sectional surface is moved based on centering the position of the target object shown in the second image. Image acquisition of all cross-sectional surfaces on the specimen is captured more efficiently without repeating image acquisition for the same cross-sectional surface.

[0021] In this embodiment, a third image corresponding to the specimen is obtained. The third image shows a third cross-sectional surface of the specimen. The third image is captured by the microscope after moving the microscope's field of view relative to the specimen according to the movement vector. A set of images for tracking the target object is compiled. The set of images includes the second and third images. Therefore, each cross-sectional surface is captured once. The set of images for tracking the object includes each captured image. It is assumed that the position of the target object on the cross-sectional surface does not change much, and the target object should remain in the field of view even if each image may not necessarily be centered on the target object. Image acquisition of all cross-sectional surfaces on the specimen is captured more efficiently without repeating image acquisition for the same cross-sectional surface.

[0022] In this embodiment, a first image corresponding to the specimen is displayed at a first time interval. A second image corresponding to the specimen is displayed at a second time interval following the first time interval. Thus, the method is applied to specimens at different time intervals, and continuous images of the specimen are obtained. When the specimen can change over time, the target object is tracked using these continuous images of the specimen.

[0023] In an embodiment, determining a specific object of interest (POI) in the segmented image that has the highest similarity score to a target object in the first image as the target object in the segmented image includes determining a first location in the first image where the specific POI is closest. Alternatively, determining a specific POI in the segmented image that has the highest similarity score to a target object in the first image includes determining a first shape of the specific POI that is most similar to a second shape of the target object in the first image. Thus, a similarity score between an POI in a segmented image and a target object in another image can be determined based on one or more factors, such as object location and / or object shape.

[0024] It should be noted that the first image can be captured using a microscope. Alternatively, the first image can be a segmented version of another image captured using a microscope. It should also be noted that image segmentation techniques can utilize artificial neural networks (ANNs).

[0025] Furthermore, to achieve the objectives described above, the present invention provides a method according to technical solution 12. The method according to technical solution 12 includes the step of obtaining a first image corresponding to a specimen. A first vector set connecting multiple target objects shown in the first image is identified. A first position in the first image corresponding to the multiple target objects is identified. A second image corresponding to the specimen (captured via a microscope) is then obtained. Image segmentation techniques are applied to the second image to obtain a segmented image. The segmented image indicates (a) a first set of segments corresponding to objects of interest and (b) a second set of segments not corresponding to any object of interest. Subgroups of objects of interest shown in the segmented image are identified. Vector sets corresponding to the subgroups of objects of interest shown in the segmented image are determined. A specific vector set in the vector set associated with the minimum difference of the first vector set is determined. A specific subgroup of objects of interest in the subgroups of objects of interest connected by the specific vector set is identified as multiple target objects. A second position in the segmented image corresponding to the multiple target objects is identified. A movement vector from the first position in the first image to the second position in the segmented image is determined. The field of view of the microscope is moved relative to the specimen according to the movement vector to capture a third image corresponding to the specimen. In an embodiment, the method may be performed by at least one device including a hardware processor. Using the method described above, by cutting thin slices from the surface of the specimen, it is possible to accurately identify and track the target object through different cross-sectional surfaces of the specimen, as described above. Furthermore, using the method described above, it is possible to accurately identify and track the target object through the specimen as it changes over time. Accuracy is increased because the target object is not identified directly from the captured image; rather, it is identified from a segmented image generated based on the captured image. The segmented image is used to filter out objects of no interest shown in the captured image. The segmented image therefore involves less "noise" in the process of identifying the target object. Based on the method described above, the target object remains within the microscope's field of view.

[0026] In this embodiment, a first image corresponding to the specimen is displayed at a first time interval. A second image corresponding to the specimen is displayed at a second time interval following the first time interval. Thus, the method is applied to specimens at different time intervals, and continuous images of the specimen are obtained. When the specimen can change over time, the target object is tracked using these continuous images of the specimen.

[0027] In this embodiment, after moving the microscope's field of view relative to the sample according to the movement vector, a third image corresponding to the sample is captured. The third image shows the sample at a third time interval following the second time interval. Thus, an image corresponding to the sample is obtained for each time interval. The target object is tracked through continuous images of the sample.

[0028] In this embodiment, a third image corresponding to the specimen is obtained. The third image shows the specimen at a third time interval following the second time interval. The third image is captured by the microscope after moving the microscope's field of view relative to the specimen according to a motion vector. A set of images tracking multiple target objects is compiled. The set of images includes the second and third images. Thus, an image corresponding to the specimen is obtained for each time interval. Target objects are tracked using consecutive images of the specimen. The set of images tracking multiple target objects comprises each image captured at each time interval.

[0029] According to one aspect of the invention, a non-transitory computer-readable medium according to claim 15 is provided. The non-transitory computer-readable medium includes instructions that, when executed by one or more hardware processors, cause operations as described in any of the above embodiments to be performed. Advantages have been discussed above.

[0030] According to another aspect of the present invention, a system according to claim 16 is provided. The system includes at least one means comprising a hardware processor. The system is configured to perform operations as described above with respect to the method according to the present invention. In embodiments, the system may include one or more components for performing the operations described above.

[0031] According to another aspect of the present invention, a system according to claim 17 is provided. The system includes one or more components for performing the operations described above.

[0032] The methods described in this section are permissible but not necessarily methods that were previously conceived or adopted. Therefore, unless otherwise stated, no method described in this section should be assumed to qualify as prior art simply by virtue of its inclusion in this section. Attached Figure Description

[0033] The embodiments are shown in the accompanying drawings by way of example and not by way of limitation. It should be noted that references to "a" or "an" embodiment in this disclosure do not necessarily refer to the same embodiment, but rather to at least one embodiment. In the accompanying drawings:

[0034] Figure 1A An example object tracking image acquisition system according to one or more embodiments is shown;

[0035] Figure 1B An example processing flow of an object tracking image acquisition system according to one or more embodiments is shown;

[0036] Figure 1C Example specimens according to one or more embodiments are shown;

[0037] Figures 2A to 2BA set of example operations for obtaining a 3D sample rendering of a tracking target object according to one or more embodiments are shown;

[0038] Figures 3A to 3B A set of example operations for obtaining continuous sample images of a tracking target object, according to one or more embodiments, are shown;

[0039] Figure 4 An example cross-section of a specimen according to one or more embodiments is shown;

[0040] Figures 5A to 5J Examples of obtaining 3D sample renderings of a tracking target object are shown according to one or more embodiments;

[0041] Figure 6 An example specimen is shown as a change over time according to one or more embodiments;

[0042] Figures 7A to 7H Examples of obtaining continuous sample images of a tracking target object are shown according to one or more embodiments;

[0043] Figures 8A to 8C A set of example operations are shown according to one or more embodiments for obtaining a 3D sample rendering that tracks multiple target objects;

[0044] Figures 9A to 9C A set of example operations for obtaining sequential specimen images of tracking multiple target objects, according to one or more embodiments, are shown;

[0045] Figures 10A to 10N Examples of obtaining sequential sample images for tracking multiple target objects are shown according to one or more embodiments; and

[0046] Figure 11 A block diagram illustrating a computer system according to one or more embodiments is shown. Detailed Implementation

[0047] In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some instances, well-known structures and devices are described in block diagram form to avoid unnecessarily obscuring the invention.

[0048] 1. Overview

[0049] 2. Object Tracking Image Acquisition System Architecture

[0050] 3. Object tracking based on comparative distances associated with objects of interest in segmented images.

[0051] 4. Example Implementation

[0052] 5. Object tracking based on comparison connection vectors associated with objects of interest in segmented images

[0053] 6. Example Implementation

[0054] 7. Hardware Overview

[0055] 8. Miscellaneous; extensions.

[0056] 1. Overview

[0057] Tracking one or more target objects using a set of specimen images. The set of images may include images across a cross-sectional surface of the specimen. Alternatively, the set of images may include sequentially captured images of the specimen over time.

[0058] One or more embodiments involve tracking a target object based on a comparative distance associated with an object of interest in a segmented image. A first image corresponding to the specimen is obtained. A first location in the first image corresponding to the target object is identified. A second image corresponding to the specimen is then obtained (captured via a microscope). Image segmentation techniques are applied to the second image to obtain a segmented image. The segmented image contains (a) a first set of segments corresponding to the object of interest and (b) a second set of segments not corresponding to any object of interest. A specific object of interest among the objects of interest shown in the segmented image, associated with the highest similarity score to the target object shown in the first image, is identified as the target object shown in the segmented image. A second location in the segmented image corresponding to the target object is identified. A movement vector from the first location in the first image to the second location in the segmented image is determined. The microscope's field of view is moved relative to the specimen according to the movement vector to capture a third image corresponding to the specimen.

[0059] One or more embodiments involve tracking target objects based on comparison connection vectors associated with objects of interest in segmented images. A first image corresponding to the specimen is obtained. A first set of vectors connecting multiple target objects shown in the first image is identified. A first position in the first image corresponding to the multiple target objects is identified. A second image corresponding to the specimen is then obtained (captured via a microscope). Image segmentation techniques are applied to the second image to obtain a segmented image. The segmented image indicates (a) a first set of segments corresponding to objects of interest and (b) a second set of segments not corresponding to any object of interest. Subgroups of objects of interest shown in the segmented image are identified. Vector sets corresponding to the subgroups of objects of interest shown in the segmented image are determined. A specific set of vectors in the vector set associated with the minimum difference of the first vector set is determined. Specific subgroups of objects of interest in the subgroups of objects of interest connected by the specific vector set are identified as multiple target objects. A second position in the segmented image corresponding to the multiple target objects is identified. A movement vector from the first position in the first image to the second position in the segmented image is determined. The microscope's field of view is moved relative to the specimen according to the movement vector to capture a third image corresponding to the specimen.

[0060] One or more embodiments described in this specification and / or listed in the claims may not be included in this overview section.

[0061] 2. Object Tracking Image Acquisition System Architecture

[0062] Figure 1A An example object tracking image acquisition system according to one or more embodiments is shown. Figure 1A As shown, system 100 includes a sample 102, a microscope 104, an artificial neural network (ANN) 108, a microscope positioning manager 114, and a data storage 116. In one or more embodiments, system 100 may include a sample 102, a microscope 104, an artificial neural network (ANN) 108, a microscope positioning manager 114, and a data storage 116. Figure 1A The components shown may have more or fewer components. Figure 1A The components shown can be local to each other or far apart from each other. Figure 1A The components shown can be implemented in software and / or hardware. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined within a single application and / or machine. An operation described relative to one component can be performed alternatively by another component.

[0063] In one or more embodiments, sample 102 is an object (or part of an object) to be studied and / or analyzed. In the life sciences, examples of samples include tissues, cells, protein structures, and viruses. In the natural resources field, examples of samples include oils, gases, organic materials, and rocks. In the semiconductor field, examples of samples include crystalline materials and semiconductor materials. Samples from additional and / or alternative fields may be used.

[0064] Sample 102 may have a certain depth. (Reference) Figure 1C The “depth” of the specimen refers to the distance from the top surface 120 to the bottom surface 122. The cross-sectional surface 124 (also referred to as a “block”) is a plane of the specimen 102, created by dividing the specimen 102 from one side to the opposite side. The cross-section 126 is a plane of the specimen 102, created by dividing the specimen 102 from the top to the bottom. The microscope 104 can capture images of the cross-sectional surface 124 from the top surface 120 to the bottom surface 122 of the specimen.

[0065] Specimen 102 may change over time. This change may be caused by stimuli. For example, various objects within the specimen may change size, shape, and / or position over time. Various objects may be formed in the specimen over time. Various objects may disappear from the specimen over time.

[0066] Sample 102 contains one or more objects of interest. The object of interest is associated with the type of object being studied. The remainder of the sample provides context for the object of interest. As an example, the sample could be a brain structure containing cell membranes and other structures. Cell membranes can be a type of object being studied. Therefore, cell membranes can be considered objects of interest. Meanwhile, other structures are not objects of interest. Other instances of objects of interest include mitochondria, specific neuronal structures, other parts of biological structures, defects within crystal structures, boundaries of crystal structures, structures within nanowires, and bubbles within liquids.

[0067] Specimen 102 contains one or more target objects. A target object is a specific object of interest. In the case of generating a 3D rendering of the specimen, the target object is the object of interest that will be observed through a cross-section of specimen 102. Alternatively, in the case of generating continuous images of a varying specimen, the target object is the object of interest that will be observed over a specific time period. Therefore, a target object can be a subset of objects of interest. As an example, the specimen may contain ten cell membranes. Cell membranes can be the type of object being studied. Therefore, all ten cell membranes can be objects of interest. However, it is possible that only one of the ten cell membranes needs to be tracked through a cross-section of the specimen. The tracked object of interest is called the target object.

[0068] Return to reference Figure 1AIn one or more embodiments, microscope 104 is an optical instrument used to better view objects that are difficult to see with the naked eye. Types of microscopes include transmission electron microscopes (TEM), scanning electron microscopes (SEM), scanning transmission electron microscopes (STEM), focused ion beam microscopes (FIB microscopes), and scanning optical microscopes. As used herein, the term "microscope" may also encompass detection systems such as energy dispersive spectroscopy (EDS) and / or electron backscatter diffraction (EBSD), wavelength dispersive spectroscopy (WDS), micro X-ray fluorescence (micro XRF), and / or X-ray metrology.

[0069] In one or more embodiments, the field of view of microscope 104 refers to a portion of sample 102 that can be viewed through the lens of microscope 104. Microscope 104 is associated with one or more positioning parameters 106 for positioning the field of view of microscope 104. The field of view of microscope 104 can be moved relative to sample 102 in various ways. Positioning parameter 106 can be, for example, the position of the electron beam within microscope 104. Alternatively or alternatively, positioning parameter 106 can be the position of the sample stage relative to microscope 104. The sample stage on which sample 102 is mounted is movable. Alternatively or alternatively, positioning parameter 106 can be the position of sample 102 relative to the sample stage. Sample 102 can be mounted at different positions on the sample stage. Alternatively or alternatively, positioning parameter 106 can be the position of the lens of microscope relative to microscope 102. Additional or alternative positioning parameters 106 can also be used.

[0070] In one or more embodiments, microscope 104 is associated with one or more microscope parameters. Microscope parameters are configurable properties of microscope 104. Examples of microscope parameters for SEM include: accelerating voltage (the voltage at which electrons are accelerated as they pass through an electron optics column); electron beam convergence angle; beam current; spot size (the diameter of the beam spot on the sample); residence time; and resolution.

[0071] Different values ​​of various microscope parameters result in images of varying quality and properties. For example, higher magnification requires a smaller spot size. Higher signal-to-noise ratio and contrast resolution require a larger beam current. However, reducing the spot size also reduces the beam current.

[0072] In one or more embodiments, the data repository 116 is any type of storage unit and / or device for storing data (e.g., a file system, database, table collection, or any other storage mechanism). Additionally, the data repository 116 may comprise multiple different storage units and / or devices. These multiple different storage units and / or devices may be of the same type or different types, or located in the same physical location. Furthermore, the data repository 116 may be implemented or executed on the same computing system as the microscope 104, ANN 108, and / or merging module 114. Alternatively or additionally, the data repository 104 may be implemented or executed on a separate computing system from the microscope 104, ANN 108, and / or merging module 114. The data repository 104 may be communicatively connected to the microscope 104, ANN 108, and / or merging module 114 via a direct connection or via a network.

[0073] Information describing the captured specimen image 116 and the segmented specimen image 118 can be implemented on any component within the system 100. However, for clarity and explanation purposes, this information is shown within the data storage 116.

[0074] In one or more embodiments, the captured specimen image 116 (also referred to herein as a "specimen image" or "image") is an image of the specimen 102 produced by the microscope 104. The captured specimen image 116 shows the area of ​​the specimen 102 within the field of view of the microscope 104. The captured specimen image 116 may be associated with a specific level of image quality. Image quality may be defined by, for example, resolution, magnification, the number of pixels in the specimen image, the pixel size in the specimen image, and / or other attributes.

[0075] In an embodiment, a set of captured sample images 116 corresponding to sample 102 is obtained. The set of captured sample images 116 may constitute a 3D rendering for tracking one or more target objects within sample 102. Alternatively, the set of captured sample images 116 may constitute a series of images for tracking one or more target objects that change and / or move within sample 102. The set of captured sample images 116 may be compiled to generate animation and / or video.

[0076] In one or more embodiments, the segmented sample image 118 (also referred to herein as a “segmented image”) is a simplified version of the captured sample image 116. The segmented sample image 118 divides the captured sample image 116 into multiple segments. Each pixel or location within the segmented sample image belongs to one of the multiple segments. Each segment is associated with one of a set of defined labels. Any number of labels may be included in the labels defined by the set.

[0077] As an example, a segmented specimen image can contain two types of segments: segments showing the object of interest, and segments not showing any object of interest. Each pixel or location within a segmented specimen image can belong to one of the segments. Therefore, a segmented specimen image can also be called a binary image, where each pixel or location of the segmented specimen image is labeled with one of two possible values.

[0078] In this embodiment, the segmented specimen image 116 comprises multiple segments, each segment displaying one or more objects of interest. As described above, each object of interest is associated with the type of object being studied. However, only a subset of the multiple objects of interest can be identified as the target object. Not all objects of interest are tracked in the 3D rendering of the specimen or in successive specimen images taken over time. Only the target object is tracked in the 3D rendering of the specimen or in successive specimen images taken over time.

[0079] In one or more embodiments, ANN 108 (also known as a deep learning system) is a machine learning computing system. ANN 108 comprises a collection of connection units or nodes called artificial neurons. Each connection transmits a signal from one artificial neuron to another. Artificial neurons process the received signals to transmit another signal to another artificial neuron. Artificial neurons can be clustered into layers. Different layers can perform different kinds of transformations on their inputs.

[0080] One type of ANN is the Convolutional Neural Network (CNN). CNNs are designed to process data that appears as multiple arrays, such as a color image composed of three two-dimensional arrays containing pixel intensities from three color channels. A typical architecture of a traditional neural network is constructed as a series of stages. The first few stages consist of two types of layers: convolutional layers and pooling layers. Convolutional layers apply a convolution operation to the input and pass the result to the next layer. Convolution simulates the response of an individual neuron to a visual stimulus. Pooling layers combine the outputs of a cluster of neurons from one layer into a single neuron in the next layer. For example, Largest Exchange setUse the maximum value of each neuron in the cluster of neurons in the previous layer. An example of a conventional neural network is u-net. u-net is in Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, pp. 234-241 (2015), which is hereby incorporated by reference.

[0081] In one or more embodiments, ANN 108 is configured to detect and / or identify objects of interest shown in a specimen image. For each location within the specimen image, ANN 108 determines the probability that the location represents an object of interest. This probability may also be referred to as a "predicted value". Prediction mask 110 indicates the predicted value for each location in the specimen image. Each location of prediction mask 110 maps to a corresponding location within the specimen image.

[0082] Based on the prediction mask 110, a segmented sample image 118 can be generated. As an example, the prediction mask can be thresholded. Locations with predicted values ​​above the threshold can be associated with a label (e.g., a segment corresponding to the object of interest). Locations with predicted values ​​below the threshold can be associated with another label (e.g., a segment not corresponding to the object of interest). Therefore, ANN 108 can be configured to perform image segmentation on the captured sample image 116 to generate the segmented sample image 118.

[0083] In one or more embodiments, additional or alternative methods may be used to perform image segmentation on the captured sample image 116 to generate a segmented sample image 118. As an example, a thresholding method may be applied to a grayscale image. Values ​​in a grayscale image within a certain range may be classified as associated with a specific label. Values ​​in a grayscale image within another range may be classified as associated with another label. As another example, a clustering method may be used. K The mean algorithm is used to segment an image into... K An iterative clustering technique can be used. As another example, edge detection methods can be used. As another example, graph segmentation methods can be used. As yet another example, watershed transform methods can be used.

[0084] In one or more embodiments, microscope positioning manager 112 refers to hardware and / or software configured to move the field of view of microscope 104 relative to specimen 102 to track one or more target objects in specimen 102. Target objects can be identified from a plurality of objects of interest shown in segmented specimen images 118. References below... Figures 2A to 2B and Figures 3A to 3B Describe an instance of an operation used to track a target object.

[0085] In this embodiment, the microscope positioning manager 112 is implemented on one or more digital devices. The term "digital device" generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablets, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general-purpose machines, function-specific hardware devices, mainframes, televisions, content receivers, printers, mobile phones, smartphones, and / or personal digital assistants (PDAs).

[0086] Figure 1B An example processing flow of an object tracking image acquisition system according to one or more embodiments is shown. Figures 1A to 1B Components labeled with the same number in the text refer to similar components.

[0087] As shown, microscope 104 is configured with a first set of values ​​for one or more positioning parameters 106. Microscope 104 is used to observe specimen 102 using the first set of values ​​for positioning parameters 106. Microscope 104 generates a currently captured specimen image 116a of specimen 102.

[0088] The currently captured sample image 116a is input into ANN 108. ANN 108 identifies objects of interest within the currently captured sample image 116a. Segmented sample images 118 are generated based on the currently captured sample image 116a. The segmented sample images 118 indicate segments of the currently captured sample image 116a corresponding to objects of interest and segments of the currently captured sample image 116a that do not correspond to any objects of interest.

[0089] The segmented specimen image 118 is input to the microscope positioning manager 112. The microscope positioning manager 112 identifies one or more target objects from the objects of interest indicated in the segmented specimen image 118. The microscope positioning manager 112 determines a movement vector from (a) the position of the target object in a previously captured specimen image to (b) the position of the target object in the segmented specimen image 118. The microscope positioning manager 112 configures a second set of values ​​for the positioning parameters 106 of the microscope 104 to move the field of view according to the movement vector. The field of view is moved relative to the specimen 102 according to the movement vector.

[0090] Microscope 104 is used to observe specimen 102 using the second set of values ​​of positioning parameters 106. Microscope 104 produces the next captured specimen image 116b of specimen 102.

[0091] The currently captured sample image 116a and the next captured sample image 116b may belong to different cross-sectional surfaces of the sample 102. Alternatively, the currently captured sample image 116a and the next captured sample image 116b may belong to samples that vary with time 102 at different time intervals.

[0092] Optionally, repeatedly captured specimen images (not shown) may be taken between the currently captured specimen image 116a and the next captured specimen image 116b. The repeatedly captured specimen images and the currently captured specimen image 116a belong to the same cross-sectional surface of specimen 102 and are taken with different values ​​of positioning parameter 106. Repeatedly captured specimen images are taken after the field of view has been moved according to the movement vector. The field of view is not moved between taking the repeatedly captured specimen image and the next captured specimen image 116b.

[0093] In one embodiment, compared to the currently captured sample image 116a, a recaptured sample image can be taken with reduced quality parameters, such as reduced resolution or reduced dwell time. Alternatively, compared to the recaptured sample image, the currently captured sample image 116a can be taken with reduced quality parameters. Using reduced quality parameters can speed up the process and / or reduce sample damage.

[0094] The process can be repeated with the next captured sample image 116b. The next captured sample image 116b is then input into ANN 108. The ANN generates segmented sample images based on the next captured sample image 116b, as described above.

[0095] A segmented specimen image based on the next captured specimen image 116b is input to the microscope positioning manager 112. The microscope positioning manager 112 identifies one or more target objects from the objects of interest indicated in the segmented specimen image based on the next captured specimen image 116b. The microscope positioning manager 112 determines a movement vector from (a) the position of the target object in the currently captured specimen image 116a to (b) the position of the target object in the segmented specimen image based on the next captured specimen image 116b. The microscope positioning manager 112 configures a third set of values ​​for the positioning parameters 106 of the microscope 104 to move the field of view according to the movement vector. The field of view is moved relative to the specimen 102 according to the movement vector.

[0096] Therefore, multiple sample images are generated by microscope 104. Information about the position of the target object in the currently captured sample image 116a is fed back into the system to generate the next captured sample image 116b. The positioning parameters 106 of microscope 104 are configured to track the target object.

[0097] 3. Object tracking based on comparative distances associated with objects of interest in segmented images.

[0098] Figures 2A to 2B A set of example operations for obtaining a 3D sample rendering of a tracking target object, according to one or more embodiments, are shown. Figures 2A to 2B One or more operations shown can be modified, rearranged, or omitted together. Therefore, Figures 2A to 2B The specific order of operations shown should not be construed as limiting the scope of one or more embodiments.

[0099] One or more embodiments include obtaining a captured image of a first cross-sectional surface of the specimen (operation 202). A microscope is configured using a first set of values ​​for one or more positioning parameters. The specimen is mounted on a stage for observation under the microscope. The first cross-sectional surface of the specimen is observed using the microscope. A captured image is generated showing a specific area of ​​the first cross-sectional surface within the microscope's field of view. An object tracking image acquisition system obtains the captured image of the first cross-sectional surface from the microscope.

[0100] One or more embodiments include identifying target objects in a captured image (operation 204). The object tracking image acquisition system obtains information identifying target objects in the captured image. This information may be entered by a user. Alternatively, the information may be generated by another application.

[0101] One or more embodiments include identifying the location of a target object in a captured image (operation 206). The object tracking image acquisition system identifies the location in a captured image corresponding to a target object. The location in the image corresponding to the target object may also be referred to herein as the "target object location in the image". The target object location in a particular image may be represented as a pair of coordinates. As an example, the target object location (3, 2) may indicate that the target object location is 3 units to the right of the lower left corner of the image and 2 units above the lower left corner of the image.

[0102] The location of a target object in a specific image can be determined based on various rules. For example, a rule can specify that the center of a target object shown in the image is considered the target object's location in the image. As another example, a rule can specify that the upper right point of a target object shown in the image is considered the target object's location in the image. As yet another example, if multiple target objects exist, the rule can specify that the centers of all multiple target objects shown in the image are considered the target object's location in the image. For instance, a specific image may contain two target objects of the same size. One target object may be centered at position (2,4). The other target object may be centered at position (4,4). Then, the centers of the two target objects may be located at (3,4). Therefore, position (3,4) can be considered the target object's location in the specific image.

[0103] One or more embodiments include determining a motion vector from the center of the captured image to the location of a target object in the captured image (operation 208). The motion vector is drawn from the center of the captured image to the location of the target object in the captured image. The motion vector may be represented as a magnitude and a direction. Alternatively or additionally, the motion vector may be represented as... x The number of units in the axis and y The number of units in the axis.

[0104] One or more embodiments include moving the microscope's field of view relative to the specimen according to a motion vector (operation 210). An object tracking image acquisition system moves the microscope's field of view according to the motion vector. The object tracking image acquisition system can transmit instructions to the microscope instructing the microscope's field of view to move according to the motion vector. The field of view can move with the same magnitude and direction as the motion vector. Alternatively, the field of view can move within a specific range of the magnitude of the motion vector and within a specific range of the direction of the motion vector. Based on the instructions, the microscope is reconfigured using another set of values ​​for positioning parameters.

[0105] In this embodiment, the position of the electron beam is moved by changing the deflector settings and / or electromagnetic settings of the lens. To move the field of view relative to the sample according to the movement vector, the electron beam moves with the same magnitude and direction as the movement vector. New values ​​can be assigned to the positioning parameters such that the electron beam moves with the same magnitude and direction as the movement vector.

[0106] In one embodiment, a stage on which the sample is mounted is moved. To move the field of view relative to the sample according to the movement vector, the stage moves by the same amount as the movement vector but in the opposite direction. New values ​​can be assigned to the positioning parameters such that the stage moves by the same amount as the movement vector but in the opposite direction.

[0107] In this embodiment, the specimen is moved and mounted at different positions on the stage. To move the field of view relative to the specimen according to the movement vector, the specimen is moved in the same magnitude and direction as the movement vector. New values ​​can be assigned to the positioning parameters such that the specimen moves in the same magnitude and direction as the movement vector.

[0108] In this embodiment, the lens of the moving microscope is moved. To move the field of view relative to the sample according to the movement vector, the lens moves in the same magnitude and direction as the movement vector. New values ​​can be assigned to the positioning parameters such that the lens moves in the same magnitude and direction as the movement vector.

[0109] In this embodiment, a combination of movements of different components of the microscope can be performed. The electron beam, stage, sample, and / or lens can be moved individually. Based on the movement vector, the combined movement shifts the field of view relative to the sample.

[0110] One or more embodiments include obtaining another captured image of the first cross-sectional surface of the specimen centered on the target object (operation 212). After moving the microscope's field of view, the position corresponding to the target object is now centered in the field of view. The object-tracking image acquisition system causes the microscope to capture another captured image of the first cross-sectional surface. Both captured images obtained at operations 202 and 212 show the first cross-sectional surface. However, the captured image obtained at operation 202 is not necessarily centered on the target object. The captured image obtained at operation 212 is substantially centered on the target object.

[0111] In this embodiment, operation 212 is not performed. Even if the captured image obtained at operation 202 is not necessarily centered on the target object, the captured image obtained at operation 202 is accepted as an image of the first cross-sectional surface for 3D rendering of the specimen. Skipping operation 212 reduces the amount of time and / or resources required to obtain 3D rendering of the specimen. Moving the microscope's field of view at operation 210 is not to obtain a captured image of the current cross-sectional surface centered on the target object, but rather to obtain a captured image of the next cross-sectional surface that tracks the target object.

[0112] One or more embodiments include setting a second cross-sectional surface as the current cross-sectional surface in preparation for obtaining the next captured image (operation 214). The second cross-sectional surface is prepared for observation under a microscope.

[0113] In this embodiment, bulk scanning electron microscopy is used. The process begins by mounting the sample on the stage of the microscope. The bulk surface of the sample is observed. The first observed bulk surface constitutes a first cross-sectional surface captured at operation 202. A thin slice is then cut from the bulk surface of the sample. Removing the thin slice removes the first cross-sectional surface and exposes a second cross-sectional surface. Thus, the second cross-sectional surface is ready for observation under the microscope.

[0114] In this embodiment, continuous cross-sectional scanning electron microscopy is used. The process begins by slicing the sample into thin sections. The first section is mounted on the stage of the microscope. The first section is observed. The surface of the first section forms the first cross-sectional surface captured at operation 202. The first section is then removed from the stage. A second section is mounted on the stage. Thus, the surface of the second section, which forms the second cross-sectional surface, is ready for observation under the microscope.

[0115] As described above at operation 226, the field of view can be moved without moving the specimen relative to the stage. In this case, the first and second slices are mounted in the same position on the microscope stage. As an example, the first slice can be mounted such that its center is located at the center of the stage. Therefore, the surface of the first slice, which serves as the first cross-sectional surface, can be observed. The field of view can then be moved by moving the stage and / or moving the lens of the microscope. When preparing the second cross-sectional surface for observation, the first slice can be removed from the stage. The second slice can be mounted such that its center is also located at the center of the stage. Therefore, the first and second slices are mounted in the same position on the microscope stage.

[0116] As described above at operation 226, the field of view can be moved by moving the specimen relative to the stage. In this case, the distance between the positions of the first slice and the second slice is equal to the distance the specimen should be moved relative to the stage. As an example, the first slice can be mounted such that the center of the first slice is located at the center of the stage. Therefore, the surface of the first slice, which is the first cross-sectional surface, can be observed. The movement vector can be determined as (4, 2). The field of view can be moved by moving the specimen relative to the stage. x Move 4 units in the direction and in y The first slice can be moved two units in the direction of observation. When preparing the second cross-sectional surface for observation, the first slice can be removed from the stage. The second slice can be mounted such that its center is located at the center of the first slice. x The direction is 4 units, and it starts from the center of the first slice. y The distance in the direction is 2 units. Therefore, the distance between the positions of the first and second slices is equal to the distance the sample should move relative to the stage.

[0117] One or more embodiments include obtaining a captured image of the current cross-sectional surface of the specimen (operation 216). The current cross-sectional surface of the specimen is observed using a microscope. A captured image is generated showing a specific area of ​​the current cross-sectional surface within the microscope's field of view. The object tracking image acquisition system obtains the captured image of the current cross-sectional surface from the microscope.

[0118] One or more embodiments include generating a segmented image based on the captured image (operation 218). The captured image obtained at operation 216 is analyzed to generate the segmented image.

[0119] In an embodiment, an ANN is applied to a captured image to generate a segmented image. The ANN receives the captured image as input. The ANN determines a predicted value for each location within the captured image. As an example, a u-net convolutional neural network can be applied to the captured image to identify neuronal structures within the captured image. The u-net convolutional network can determine the probability that each location within the captured image displays at least a portion of a specific neuronal structure. The corresponding predicted values ​​for the locations within the captured image constitute a prediction mask.

[0120] A thresholding algorithm is applied to the prediction mask. A fixed threshold may exist for each location in the prediction mask. Alternatively, the threshold can vary across locations within the prediction mask. The threshold at a particular location can be adjusted based on predicted values ​​corresponding to, for example, surrounding locations and / or other factors.

[0121] The predicted value for each location is compared to a threshold for that location. If the predicted value is higher than the threshold, the corresponding location in the captured image is determined to display an object of interest. Conversely, if the predicted value is lower than the threshold, the corresponding location in the captured image is determined to not display any object of interest. Therefore, each location in the captured image is determined to either display an object of interest or not display any object of interest. This generates a segmented image. The segmented image displays two types of segments: (a) segments containing locations in the captured image that display objects of interest and (b) segments containing locations in the captured image that do not display any objects of interest.

[0122] In embodiments, as described above, additional or alternative methods may be used to perform image segmentation on the captured image to generate a segmented image.

[0123] One or more embodiments include identifying objects of interest (POIs) shown in a segmented image, which are associated with the highest similarity score of a target object shown in a captured image of a previous cross-sectional surface (operation 220). Each individual segment in the segmented image marked as displaying an POI is determined as a separate POI. The similarity score between an POI shown in a segmented image and a target object shown in a captured image of a previous cross-sectional surface can be determined in various ways.

[0124] In this embodiment, a similarity score is determined based on the object's location. The object of interest whose location in the captured image is closest to the target object in the previous cross-sectional surface is associated with the highest similarity score.

[0125] Identify the corresponding location in the segmented image for each object of interest. The above-described operation 206 is an example of an operation for identifying the location in a specific image corresponding to a particular object. The same set of rules used in operation 206 to determine the location of the target object in the captured image of the first cross-sectional surface is used to determine the corresponding location in the segmented image corresponding to the object of interest.

[0126] As an example, a rule can specify that the center of an object shown in an image is considered the location of the target object in the image. By applying the rule, the center of the target object shown in the captured image of a first cross-sectional surface is considered the location of the target object in the captured image. By applying the same rule, the center of the object of interest shown in a segmented image of a specific cross-sectional surface is considered the location corresponding to the object of interest in the segmented image.

[0127] For each object of interest, the object tracking image acquisition system determines the distance between (a) the target object's location in the captured image of the previous cross-sectional surface and (b) the location in the segmented image of the current cross-sectional surface corresponding to the object of interest. By repeating this calculation for each object of interest, the object tracking image acquisition system thus determines a set of distances. The object tracking image acquisition system identifies the minimum distance in this set of distances. The specific object of interest corresponding to the minimum distance is the object of interest closest to the target object's location in the captured image of the previous cross-sectional surface. The object tracking image acquisition system identifies this specific object of interest as the target object.

[0128] As an example, the target object location in the captured image of the previous cross-sectional surface could be (3, 2). The segmented image of the current cross-sectional surface can contain two objects of interest. The location in the segmented image corresponding to the first object of interest could be (4, 5). The location in the segmented image corresponding to the second object of interest could be (6, 7). The distance between (a) the target object location in the captured image of the previous cross-sectional surface and (b) the location in the segmented image of the current cross-sectional surface corresponding to the first object of interest is (4, 5) - (3, 2) = = 3.16. (a) The distance between the target object location in the captured image of the previous cross-sectional surface and (b) the location in the segmented image of the current cross-sectional surface corresponding to the second object of interest is (6, 7) - (3, 2) = = 5.83. The minimum distance is 3.16, which corresponds to the first object of interest. Therefore, the first object of interest can be identified as the target object.

[0129] In this embodiment, a similarity score is determined based on the object's shape. The object of interest whose shape is most similar to the target object shown in the captured image of the previous cross-sectional surface is associated with the highest similarity score.

[0130] As an example, a specific type of object of interest can be associated with a specific geometry. For instance, mitochondria can be modeled as an ellipse. Therefore, the elliptical parameters (e.g., eccentricity, major axis direction, surface) of the target object shown in a captured image of the previous cross-sectional surface can be determined. The elliptical parameters of each object of interest shown in the segmented image can be determined. The elliptical parameters of the target object and each object of interest can be compared. The object of interest associated with the smallest difference in elliptical parameters of the target object is identified as the most similar in shape and is therefore identified as the target object.

[0131] As another example, the shape of the target object shown in the captured image of the previous cross-sectional surface can be identified. The shape of each object of interest shown in the segmented image can be identified. The shape of the target object can be overlaid on top of the shape of each object of interest. The non-overlapping area between two shapes is determined for each object of interest. The object of interest associated with the smallest non-overlapping area is determined to be the most similar in shape and is therefore identified as the target object.

[0132] As another example, a bounding box (e.g., a rectangle with the shape closest to the object) can be drawn around the target object shown in the captured image of the previous cross-sectional surface. A bounding box can be drawn around each object of interest shown in the segmented image. The difference between (a) the area of ​​the bounding box around the target object and (b) the area of ​​the bounding box around each object of interest can be calculated. The object of interest associated with the smallest difference in bounding box area is determined to be the most similar in shape and is therefore identified as the target object.

[0133] As another example, the area and perimeter of the target object shown in the captured image of the previous cross-sectional surface can be determined. The area-perimeter ratio of the target object can be determined. The area and perimeter of each object of interest shown in the segmented image can be determined. The area-perimeter ratio of each object of interest can be determined. The difference between (a) the area-perimeter ratio of the target object and the area-perimeter ratio of each object of interest can be calculated. The object of interest associated with the smallest difference in area-perimeter ratio is identified as the most similar in shape and is therefore identified as the target object.

[0134] As another example, a similarity score between a target object shown in a captured image of a previous cross-sectional surface and an object of interest shown in a segmented image can be determined based on a combination of factors described above, such as ellipse parameters, non-overlapping area, bounding box area, object area, and / or perimeter. Alternatively, a similarity score between a target object shown in a captured image of a previous cross-sectional surface and an object of interest shown in a segmented image can be determined based on other factors.

[0135] In this embodiment, a similarity score is determined based on a combination of object location, object shape, and / or other factors. Each factor may be weighted equally or differently.

[0136] In an embodiment, the object tracking image segmentation system may perform operation 220 using a segmented image of the previous cross-sectional surface instead of a captured image of the previous cross-sectional surface. Therefore, the object tracking image segmentation system identifies objects of interest shown in the segmented image of the current cross-sectional surface, which are associated with the highest similarity score of the target object shown in the segmented image of the previous cross-sectional surface.

[0137] One or more embodiments include identifying the location of a target object in a segmented image of the current cross-sectional surface (operation 222). As described above at operation 220, a specific object of interest shown in the segmented image of the current cross-sectional surface is identified as a target object. An example of an operation for identifying the location in a specific image corresponding to a specific object is described above with reference to operation 206. The same set of rules used in operation 206 to determine the location of a target object in a captured image of a first cross-sectional surface is used to determine the location of the target object in the segmented image.

[0138] One or more embodiments include determining a movement vector (operation 224) from the location of a target object in a captured image of a previous cross-sectional surface to the location of a target object in a segmented image of the current cross-sectional surface. Reference above to operation 208 describes an example of an operation for determining a movement vector from one location in an image to another location in the image.

[0139] In this embodiment, the object tracking image segmentation system may use a segmented image of the previous cross-sectional surface instead of a captured image of the previous cross-sectional surface to perform operation 224. Therefore, the object tracking image segmentation system determines a movement vector from the target object position in the segmented image of the previous cross-sectional surface to the target object position in the segmented image of the current cross-sectional surface.

[0140] One or more embodiments include moving the field of view of the microscope according to a motion vector (operation 226). An example of an operation for moving the field of view of the microscope according to a motion vector was described above with reference to operation 210.

[0141] One or more embodiments include obtaining another captured image of the current cross-sectional surface of the specimen (operation 228). The above-described operation 212 describes an example of an operation in which another captured image of a particular cross-sectional surface is obtained after the microscope's field of view is moved to center on the target object in the particular cross-sectional surface.

[0142] In this embodiment, operation 228 is not performed. Even though the captured image of the current cross-sectional surface obtained at operation 216 is not necessarily centered on the target object, the captured image obtained at operation 216 is accepted as an image of the current cross-sectional surface for 3D rendering of the specimen.

[0143] One or more embodiments include determining whether any additional cross-sectional surfaces to be observed exist (operation 230).

[0144] In this embodiment, block scanning electron microscopy is used. An object tracking image acquisition system determines whether to perform another sectioning of the specimen. If sufficient depth still exists in the specimen, then another sectioning may be performed. However, if sufficient depth does not exist in the specimen, then another sectioning is not performed. Alternatively, if the number of observed cross-sectional surfaces is insufficient, then another sectioning may be performed. However, if the number of observed cross-sectional surfaces is sufficient, then another sectioning is not performed. Additional or alternative criteria may be used to determine whether to perform another sectioning to expose additional cross-sectional surfaces for observation under the microscope.

[0145] In this embodiment, continuous cross-sectional scanning electron microscopy is used. The specimen is initially cut into a set of slices. An object tracking image acquisition system determines whether there are any slices of the specimen that have not yet been observed.

[0146] If an additional cross-sectional surface exists to be observed, one or more embodiments include setting the next cross-sectional surface as the current cross-sectional surface in preparation for obtaining the next captured image (operation 232). The example of an operation for preparing the next cross-sectional surface for observation under a microscope is described above with reference to operation 214.

[0147] The process is repeated 216 to 230 regarding the current cross-sectional surface. Therefore, the object tracking image acquisition system traverses each cross-sectional surface of the specimen. After acquiring a captured image of a particular cross-sectional surface, the object tracking image acquisition system moves the microscope's field of view to prepare for acquiring a captured image of the next cross-sectional surface. The object tracking image acquisition system thus tracks the target object across multiple cross-sectional surfaces of the specimen.

[0148] After observing all cross-sectional surfaces, one or more embodiments include compiling a 3D rendering of the specimen that tracks the target object (operation 234).

[0149] In this embodiment, operations 212 and 230 are performed. At operations 212 and 230, captured images are obtained after the microscope's field of view has been moved to be centered on the target object. The captured images at operations 212 and 230 are compiled to generate a 3D rendering of the specimen. However, the captured images obtained at operations 202 and 216 are not necessarily centered on the target object. At operations 202 and 216, the captured images are obtained relative to the previous cross-sectional surface to determine how much the target object has moved. Therefore, when generating the 3D rendering of the specimen, the captured images at operations 202 and 216 are not compiled.

[0150] In this embodiment, operations 212 and 230 are not performed. Even though the captured images obtained at operations 202 and 216 are not necessarily centered on the target object, the captured images obtained at operations 202 and 216 are accepted as sufficiently tracking the target object. It is assumed that the target object does not significantly change position between the cross-sectional surfaces. Based on this assumption, the field of view for capturing images of a specific cross-sectional surface can be located based on the target object position on the previous cross-sectional surface. Therefore, the captured images at operations 202 and 216 are compiled to generate a 3D rendering of the specimen. In this embodiment, each captured image for each cross-sectional surface is compiled to generate a 3D rendering of the specimen.

[0151] In this embodiment, captured images selected for compilation are used to generate the animation. First, the captured image associated with the first cross-sectional surface is placed in the animation. Then, the captured image associated with the second cross-sectional surface is placed in the animation. Therefore, all captured images are ordered within the animation. The user can play the animation. The user interface can continuously present the ordered captured images. The animation is represented by movement through the cross-section of the specimen while tracking the target object.

[0152] In this embodiment, captured images, optionally used for compilation, are used to generate an interactive 3D model. Users can rotate, zoom in, or zoom out of the 3D model. The 3D model can track a target object through a cross-section of the specimen.

[0153] In this embodiment, additional or alternative 3D renderings may be generated.

[0154] Figures 3A to 3B A set of example operations for obtaining continuous sample images of a tracking target object, according to one or more embodiments, are shown. Figures 3A to 3B One or more operations shown can be modified, rearranged, or omitted together. Therefore, Figures 3A to 3B The specific order of operations shown should not be construed as limiting the scope of one or more embodiments.

[0155] One or more embodiments include acquiring a captured image of the sample at a first time interval (operation 302). The microscope is configured using a first set of values ​​for one or more positioning parameters. The sample is mounted on the stage for observation under the microscope. Stimuli can be applied to the sample, causing it to change over time. For example, a chemical reaction may occur within the sample. Bubbles may be generated within the sample. Various elements and / or compounds may be generated in the sample.

[0156] The surface of a specimen is observed using a microscope. Captured images are generated, showing specific areas of the specimen within the microscope's field of view. An object tracking image acquisition system obtains captured images of the specimen from the microscope.

[0157] One or more embodiments include identifying a target object in a captured image (operation 304). The foregoing reference to operation 204 describes an example of an operation for identifying a target object in a captured image.

[0158] One or more embodiments include identifying the location of a target object in a captured image (operation 306). The above description with reference to operation 206 illustrates an example of an operation for identifying the location of a target object in a captured image.

[0159] One or more embodiments include determining a movement vector (operation 308) from the center of a captured image to the location of a target object in the captured image. Reference to operation 208 describes an example of an operation for determining a movement vector from one location in an image to another location in the image.

[0160] One or more embodiments include moving the microscope's field of view relative to the specimen according to a motion vector (operation 310). The example of operation for moving the microscope's field of view according to a motion vector is described above with reference to operation 210. The field of view moves in real time. As the field of view moves, the specimen continues to change over time. It is impossible to obtain another captured image of the specimen in the first time interval because the first time interval has passed and the appearance of the specimen has changed.

[0161] One or more embodiments include acquiring captured images of the specimen at the current time interval (operation 312). The time interval between acquiring captured images of the specimen can be specified by the user and / or another application. The time interval between each captured image can be the same or different. As an example, the user can specify that a specimen image will be captured every 0.1 seconds. A first specimen image may be captured at 9:00:00.100, referred to as the "first time interval". A second time interval for capturing a second specimen image will then be 9:00:00.200. A third time interval for capturing a third specimen image will then be 9:00:00.300. As another example, the user can specify that the microscope will capture consecutive specimen images at irregular time intervals. The microscope will capture a first specimen image at 9:00:00.100, a second specimen image at 9:00:00.300, and a third specimen image at 9:00:00.400. When a time interval for capturing another specimen image occurs, the microscope captures another specimen image.

[0162] The surface of a specimen is observed using a microscope. Captured images are generated, showing specific areas of the specimen within the microscope's field of view. An object tracking image acquisition system obtains captured images of the specimen from the microscope.

[0163] One or more embodiments include generating a segmented image based on a captured image (operation 314). The example of an operation for generating a segmented image based on a captured image is described above with reference to operation 218.

[0164] One or more embodiments include identifying an object of interest shown in a segmented image, the object of interest being associated with the highest similarity score of a target object shown in a captured image in a previous time interval (operation 316). Reference above to operation 220 describes an example of an operation for identifying an object of interest shown in a segmented image, the object of interest being associated with the highest similarity score of a target object shown in another image.

[0165] In one embodiment, the captured image of the immediately preceding time interval is used. The object of interest shown in the segmented image is identified, and this object of interest is associated with the highest similarity score of the target object shown in the captured image of the immediately preceding time interval. In another embodiment, the captured image of the time interval preceding the immediately preceding time interval is used. This may be necessary due to insufficient processing speed. As an example, images may be captured every 0.1 seconds, but the processing time may be 1 second. In this case, captured images from 10 time intervals prior to the current time interval can be used.

[0166] One or more embodiments include identifying the location of a target object in a segmented image at the current time interval (operation 318). The example of an operation for identifying the location of a target object in a segmented image was described above with reference to operation 222.

[0167] One or more embodiments include determining a movement vector (operation 320) from the location of a target object in a captured image of a previous time interval to the location of a target object in a segmented image of the current time interval. Reference above to operation 208 describes an example of an operation for determining a movement vector from one location in an image to another location in the image.

[0168] One or more embodiments include moving the field of view of the microscope according to a motion vector (operation 226). An example of an operation for moving the field of view of the microscope according to a motion vector was described above with reference to operation 210.

[0169] One or more embodiments include determining whether the current time is still within a time period of interest (operation 324). The time period of interest can be specified by a user and / or another application. As an example, the user can specify that the time period of interest is 3 minutes. The user can specify that sample images will be captured every 0.5 seconds. Based on user input, consecutive sample images can be captured every 0.5 seconds over a total of 3 minutes. After 3 minutes have passed since the first sample image was captured, the time period of interest has elapsed. As another example, the user can specify that the time period of interest is the time required to capture 1000 sample images. The user can specify that sample images will be captured every 0.8 seconds. Consecutive sample images can be captured every 0.8 seconds until 1000 sample images have been captured. After the 1000th sample image has been captured, the time period of interest has elapsed.

[0170] If the time period of interest has not yet passed, then operations 312 to 322 are repeated for each time interval. Thus, the object tracking image acquisition system acquires captured images of the specimen within the time period of interest. After acquiring a captured image in each time interval, the object tracking image acquisition system moves the microscope's field of view to prepare for acquiring a captured image in the next time interval. The object tracking image acquisition system thus tracks the target object within the time intervals of interest.

[0171] After the time period of interest has passed, one or more embodiments include compiling successive sample images of the target object over time (operation 326). The captured images at operations 302 and 312 are compiled into a set of successive sample images of the target object. In an embodiment, each captured image at each time interval is compiled into the set of successive sample images.

[0172] Even if the captured images obtained at operations 302 and 312 are not necessarily centered on the target object, the captured images obtained at operations 302 and 312 are accepted as sufficiently tracking the target object.

[0173] In this embodiment, captured images are compiled to generate an animation. The captured images are ordered into the animation in chronological order based on the time when they were generated. The user can play the animation. The user interface can continuously present the ordered captured images. The animation shows the changes of the sample over time while tracking the target object.

[0174] In some embodiments, additional or alternative compilations of continuous images may be generated.

[0175] 4. Example Implementation

[0176] For clarity, detailed examples are described below. The components and / or operations described below should be understood as specific instances that may not be applicable to some embodiments. Therefore, the components and / or operations described below should not be construed as limiting the scope of any claims.

[0177] Figure 4 An example cross-section of a specimen according to one or more embodiments is shown. Specimen 400 includes a target object 410 extending through a cross-section of specimen 400. Specimen 400 also includes other objects (not shown). Various cross-sectional surfaces 402 to 404 of the specimen can be observed under a microscope. At cross-sectional surface 402, the target object 410 is located... Position X At point 404 on the cross-sectional surface, the target object 410 is located... Position Y Place.

[0178] Figures 5A to 5J Examples of obtaining 3D sample renderings of a tracking target object are shown according to one or more embodiments.

[0179] Figure 5A The cross-sectional surface 502 of the specimen is displayed. Several objects are presented on the cross-sectional surface 502, including objects of interest 552 and objects of non-interest 554. As an example, the object of interest could be mitochondria, while the objects of non-interest could be other biological structures within the specimen. A first set of values ​​for the microscope's positioning parameters produces a field of view 512 relative to the cross-sectional surface 502. Therefore, the area of ​​the cross-sectional surface 502 within the field of view 512 can be observed with the microscope, while the area outside the field of view 512 cannot be observed with the microscope.

[0180] Figure 5B Image 522 shows a captured image of cross-sectional surface 502. Image 522 shows the area of ​​cross-sectional surface 502 within the field of view 512. Image 522 also shows both the object of interest 552 and the object of no interest 554.

[0181] The user views the captured image 522. The user enters user input to identify the target object 510. Position X The center of the target object 510 is identified. Position X It can be associated with coordinates (5, 5). Position X The location of the target object is considered to be in the captured image 522.

[0182] Prepare the next cross-section surface 504 for observation under a microscope. Figure 5C The cross-sectional surface 504 of the specimen is displayed. Objects of interest 552 and objects of no interest 554, presented on the cross-sectional surface 502, are also presented on the cross-sectional surface 504. However, the objects can be presented in different shapes and / or different positions. The first set of values ​​for the positioning parameters is still used to configure the microscope. The first set of values ​​for the positioning parameters produces a field of view 514a relative to the cross-sectional surface 504. Therefore, the area of ​​the cross-sectional surface 504 within the field of view 514a can be observed through the microscope, while the area outside the field of view 514a cannot be observed through the microscope.

[0183] Figure 5D Image 524a shows a captured image of cross-sectional surface 504. Image 524a shows the area of ​​cross-sectional surface 504 within the field of view 514a. Image 524a shows both the object of interest 552 and the object of no interest 554. It should be noted that the target object 510 is not necessarily located at the center of the captured image 524a.

[0184] Figure 5E The segmented image 534 is generated based on the captured image 524a of the cross-sectional surface 504. Segmented image 534 indicates that specific segments of objects of interest 552a to 552b are displayed. Segmented image 534 indicates that no objects of interest are displayed in any area outside the specific segments.

[0185] exist Position Z The center of object of interest 552a is identified. Position Z The position is considered to correspond to the segmented image 534 of the object of interest 552a. Position Z It can be associated with coordinates (9, 9). In the captured image 522... Position X (Target object location) and segmentation image 534 Position Z The distance between them is calculated as follows = 5.66.

[0186] exist Position Y The center of object of interest 552b is identified. Position Y The position is considered to correspond to the segmented image 534 of the object of interest 552b. Position Y It can be associated with coordinates (7, 6). In the captured image 522... Position X (Target object location) and segmentation image 534 Position Y The distance between them is calculated as follows = 2.24.

[0187] Therefore, the minimum distance was determined to be 2.24. The minimum distance corresponds to the object of interest 552b. The object of interest 552b was identified as the target object 510. Position Y The location of the target object in segmented image 534 is considered.

[0188] Figure 5F Showing the location of the target object from captured image 522 ( Position X ) to the target object location in segmented image 534 ( Position Y The movement vector is 556. The movement vector can be represented as (2, 1). The movement vector is 2 units long in the x-direction and 1 unit long in the y-direction.

[0189] Figure 5G The cross-sectional surface 504 of the specimen is displayed. A second set of positioning parameters is configured for use with the microscope. This second set of positioning parameters moves the field of view according to a movement vector 556. Specifically, the distance between the previously used field of view 514a and the current field of view 514b is equal to the movement vector 556. Therefore, the area of ​​the cross-sectional surface 504 within the field of view 514b can be observed with the microscope, while the area outside the field of view 514b cannot be observed. As shown, the target object 510 is now centered in the field of view 514b.

[0190] Figure 5H Image 524b shows a captured image of the cross-sectional surface 504. Image 524b shows the area of ​​the cross-sectional surface 504 within the field of view 514b. Captured images 524a and 524b are similar, except that captured image 524b is centered on the target object 510.

[0191] Prepare the next cross-sectional surface 506 for observation under a microscope. Repeat the above process for cross-sectional surface 506. Figure 5I The cross-sectional surface 506 of the specimen is displayed. Objects of interest 552 and objects of no interest 554, presented on cross-sectional surfaces 502 and 504, are also presented on cross-sectional surface 506. However, the objects can be presented in different shapes and / or different positions. A second set of values ​​for the positioning parameters is still used to configure the microscope. This second set of values ​​for the positioning parameters produces a field of view 516a relative to the cross-sectional surface 506. Therefore, the area of ​​the cross-sectional surface 506 within the field of view 516a can be observed through the microscope, while the area outside the field of view 516a cannot be observed through the microscope.

[0192] Figure 5JImage 526a shows a captured image of cross-sectional surface 506. Image 526a shows the area of ​​cross-sectional surface 506 within the field of view 516a. Image 526a shows both the object of interest 552 and the object of no interest 554. It should be noted that the target object 510 is not necessarily located at the center of the captured image 526a.

[0193] Therefore, the process can be repeated for each cross-sectional surface of the specimen. The captured images centered on the target object 510 (e.g., captured image 524b, but not captured images 524a and 526a) are compiled to generate a 3D rendering of the specimen.

[0194] Figure 6 An example specimen is shown that varies over time according to one or more embodiments. Specimen 600 includes a target object that moves and changes over time. Specimen 600 also includes other objects (not shown). At time T1, the target object 610a may be located at... Position X At time T2, the same target object 610b can be located at... Position Y .

[0195] Figures 7A to 7H Examples of obtaining continuous sample images of a tracking target object are shown according to one or more embodiments.

[0196] Figure 7A The specimen 700 is displayed at time T1. Several objects are presented on the surface of the specimen 700, including object of interest 752 and object of no interest 754. A first set of values ​​for the microscope's positioning parameters produces a field of view 712 relative to the specimen 700. Therefore, the area of ​​the specimen 700 within the field of view 712 can be observed by the microscope, while the area outside the field of view 712 cannot be observed by the microscope.

[0197] Figure 7B Image 722, captured at time T1, shows the area of ​​sample 700 within the field of view 712. Image 722 also shows both object of interest 752 and object of no interest 754.

[0198] The user views the captured image 722. The user enters user input to identify the target object 710. Position X The center of the target object 710 is identified. Position X It can be associated with coordinates (5, 5). Position X The location of the target object is considered to be in the captured image 722.

[0199] The time interval has passed, and the current time is T2. Figure 5CThe specimen 700 is shown at time T2. The object of interest 752 and the object of no interest 754, which were presented on the specimen at time T1, are also presented on the specimen at time T2. However, the objects may be presented in different shapes and / or different positions. The first set of values ​​for the positioning parameters is still used to configure the microscope. The first set of values ​​for the positioning parameters produces a field of view 714 relative to the specimen 700. Therefore, the area of ​​the specimen 700 within the field of view 714 can be observed through the microscope, while the area outside the field of view 714 cannot be observed through the microscope.

[0200] Figure 7D Image 724, captured at time T2, shows the area of ​​specimen 700 within the field of view 714. Image 724 shows both the object of interest 752 and the object of non-interest 754. It should be noted that the target object 710 is not necessarily located at the center of the captured image 724.

[0201] Figure 7E The image 734 is a segmentation image generated at time T2 based on the captured image 724 of sample 700. Segmentation image 734 indicates specific segments showing objects of interest 752a to 752b. Segmentation image 734 indicates that no objects of interest are shown in any region outside the specific segments.

[0202] exist Position Z The center of the object of interest 752a is identified. Position Z The position is considered to correspond to the segmented image 734 of the object of interest 752a. Position Z It can be associated with coordinates (9, 9). In the captured image 722... Position X (Target object location) and segmentation image 734 Position Z The distance between them is calculated as follows = 5.66.

[0203] exist Position Y The center of object of interest 752b is identified. Position Y The position is considered to correspond to the segmented image 734 of the object of interest 752b. Position Y It can be associated with coordinates (7, 6). The captured image 722... Position X (Target object location) and segmentation image 734 Position Y The distance between them is calculated as follows = 2.24.

[0204] Therefore, the minimum distance was determined to be 2.24. The minimum distance corresponds to the object of interest 752b. The object of interest 752b was identified as the target object 710. Position Y The location of the target object in segmented image 734 is considered.

[0205] Figure 7F Showing the location of the target object in the captured image 722 ( Position X ) to the target object location in segmented image 734 ( Position Y The movement vector is 756. The movement vector can be represented as (2, 1). The movement vector is 2 units long in the x-direction and 1 unit long in the y-direction.

[0206] The time interval has elapsed, and the current time is T3. The process is repeated for sample 700 at time T3. Figure 7G The specimen 700 is shown at time T3. The object of interest 752 and the object of no interest 754, which were presented on the specimen 700 at times T1 and T2, are also presented on the specimen 700 at time T3. However, the objects may be presented in different shapes and / or different positions.

[0207] The second set of positioning parameters is configured for use with the microscope. This second set of positioning parameters moves the field of view according to the movement vector 756. Specifically, the distance between the previously used field of view 714 and the current field of view 716 is equal to the movement vector 756. Therefore, the area of ​​the specimen 700 within the field of view 716 can be observed with the microscope, while the area outside the field of view 716 cannot be observed with the microscope.

[0208] The microscope's field of view moves in real time. As the microscope's field of view moves, the sample 700 continues to change. Therefore, from time T2 to time T3, the shape and / or position of the target object 710 has changed. Therefore, even if the field of view has been updated according to the movement vector 756, the target object 710 is not necessarily located at the center of the current field of view 716.

[0209] Figure 7H Image 726, captured at time T3, shows the area of ​​specimen 700 within the field of view 716. Image 726 shows both object of interest 752 and object of non-interest 754. It should be noted that the target object 710 is not necessarily located at the center of the captured image 726.

[0210] Therefore, the process can be repeated for each time interval within the time period of interest. Each captured image (e.g., captured images 722, 724, 726) is compiled to generate an animation showing the changes of the sample over time.

[0211] 5. Object tracking based on comparison connection vectors associated with objects of interest in segmented images

[0212] Figures 8A to 8C A set of example operations are shown according to one or more embodiments for obtaining a 3D sample rendering that tracks multiple target objects. Figures 8A to 8BOne or more operations shown can be modified, rearranged, or omitted together. Therefore, Figures 8A to 8B The specific order of operations shown should not be construed as limiting the scope of one or more embodiments.

[0213] One or more embodiments include obtaining a captured image of a first cross-sectional surface of the specimen (operation 802). The foregoing reference to operation 202 describes an example of an operation for obtaining a captured image of a first cross-sectional surface of the specimen.

[0214] One or more embodiments include identifying a set of target objects in a captured image (operation 804). The foregoing reference to operation 204 describes an example of an operation for identifying target objects in a captured image.

[0215] One or more embodiments include identifying a first set of connection vectors for target objects in a captured image (operation 806). The object tracking image acquisition system identifies the location corresponding to each target object within the group. The location corresponding to each target object may be, for example, the center of each target object. Vectors may be drawn between every two target objects within the group. The set of connection vectors for the target objects contains at least a sufficient number of vectors for connecting to each target object at least once.

[0216] As an example, a set of target objects may contain Object A , Object B and Object C . Location A (2, 2) can correspond to right Elephant A . Position B (4, 2) can correspond to Object B. Location C (2, 5) can correspond to Object C .

[0217] The set of connection vectors of the target object may contain: (a) from Location A (2, 2) to Position B The vector of (4, 2); and (b) from Location A (2, 2) to Position C A vector of (2, 5). Each vector connects two target objects. The set of connection vectors is connected to... Object A , Object B and Object C Each of them must be read at least once.

[0218] Alternatively, the set of connection vectors of the target object may contain: (a) from Location A (2, 2) to Position B (b) from the vector of (4, 2); Location A (2, 2) to Position CThe vectors are (2, 5). Each vector connects two target objects; and (c) from... Bit Place B (4, 2) to Position C The vector of (2, 5).

[0219] One or more embodiments include identifying target group locations in captured images (operation 808). The object tracking image acquisition system identifies locations in captured images corresponding to target object groups. Locations in an image corresponding to a group of target objects may also be referred to herein as "target group locations in the image." Target group locations in a specific image can be expressed as a pair of coordinates. As an example, a target group location (3, 2) may indicate that the target group location is 3 units to the right of the lower left corner of the image and 2 units above the lower left corner of the image.

[0220] The location of a target group in a specific image can be determined based on various rules. For example, a rule can specify that the center of a group of target objects shown in an image is considered the target group location in the image. For instance, a specific image may contain two target objects of the same size. One target object may be centered at position (2, 4). The other target object may be centered at position (4, 4). Then, the center of the two target objects may be located at (3, 4). Therefore, position (3, 4) can be considered the target group location in the specific image. As another example, a rule can specify that the upper right point of a group of target objects shown in an image is considered the target group location in the image. As yet another example, a rule can specify that the center of the upper right point of each target object within a group is considered the target group location in the image. For instance, a specific image may contain two target objects. The upper right point of the first target object is located at position (2, 5). The upper right point of the second target object is located at position (4, 5). Then, the center of the two upper right points is located at (3, 5). Therefore, position (3, 5) can be considered the target group location in the specific image.

[0221] One or more embodiments include determining a motion vector (operation 810) from the center of the captured image to the location of a target group within the captured image. The motion vector is drawn from the center of the captured image to the location of the target group within the captured image. The motion vector may be represented as a magnitude and a direction. Alternatively or additionally, the motion vector may be represented as... x The number of units in the axis and y The number of units in the axis.

[0222] One or more embodiments include moving the field of view of the microscope relative to the sample according to a motion vector (operation 812). The example of an operation for moving the field of view of the microscope according to a motion vector is described above with reference to operation 210.

[0223] One or more embodiments include obtaining another captured image of a first cross-sectional surface of the specimen centered on a target object (operation 814). Reference above to operation 212 describes an example of an operation for obtaining another captured image of a specific cross-sectional surface after shifting the microscope's field of view to center on a set of target objects within that surface.

[0224] In this embodiment, operation 814 is not performed. Even if the captured image of the first cross-sectional surface obtained at operation 802 is not necessarily centered on the target object, the captured image obtained at operation 802 is accepted as an image of the first cross-sectional surface for 3D rendering of the specimen.

[0225] One or more embodiments include setting a second cross-sectional surface as the current cross-sectional surface in preparation for obtaining the next captured image (operation 816). Reference above to operation 214 describes an example of an operation for preparing the next cross-sectional surface for observation under a microscope.

[0226] One or more embodiments include obtaining a captured image of the current cross-sectional surface of the specimen (operation 818). An example of an operation for obtaining a captured image of the current cross-sectional surface of the specimen is described above with reference to operation 216.

[0227] One or more embodiments include generating a segmented image based on a captured image (operation 820). The example of an operation for generating a segmented image based on a captured image is described above with reference to operation 218.

[0228] One or more embodiments include identifying subgroups of objects of interest (POIs) shown in a segmented image (operation 822). Each individual segment in the segmented image, marked as displaying an POI, is determined as a separate POI. The object tracking image acquisition system identifies subgroups of POIs shown in the segmented image. Each subgroup contains a number of POIs equal to the number of target objects.

[0229] As an example, a set of target objects identified by the user may contain three objects. The segmented image can display five segments labeled as displaying objects of interest. Each of the five segments is identified as a separate object of interest, containing... object A , Object B , Object C , Object D and Object E An object tracking image acquisition system can identify subgroups of objects of interest, each subgroup containing three objects of interest. For example, a subgroup might contain... Object A , Object B , Object C Another subgroup may contain object B. Object C , Object DAnother subgroup may contain Object A , Object D , Object E .

[0230] One or more embodiments include identifying sets of connection vectors that correspond to subgroups of objects of interest in a segmented image (operation 824). For each subgroup identified at operation 822, the object tracking image acquisition system draws vectors connecting the objects within the subgroup. An example of an operation for identifying sets of connection vectors connecting a specific set of objects was described above with reference to operation 806. The number of vectors drawn at operation 824 for each subgroup of objects of interest is the same as the number of vectors drawn at operation 806 for the target object group. One or more sets of connection vectors may be drawn for each subgroup of objects of interest.

[0231] As an example, a set of target objects identified by the user may contain three target objects. The object tracking image acquisition system can draw a set of two connection vectors connecting the three target objects.

[0232] The segmented image can be displayed as five segments labeled to represent objects of interest. Each of the five segments is identified as a separate object of interest, containing... Object A , Object B , Object C , Object D and Object E . Location A (2, 2) can correspond to right Elephant A . Position B (4, 2) can correspond to Object B . Position C (2, 5) can correspond to Object C . Position D (5, 5) can correspond to Object D . Position E (5, 4) can correspond to Object E .

[0233] The object tracking image acquisition system can identify subgroups of objects of interest, with each subgroup containing three objects of interest. For example, the first subgroup might contain... Object A , Object B , Object C The second subgroup may contain Object B, Object C , Object D The third subgroup may contain Object A , Object D , Object E .

[0234] The object tracking image acquisition system can identify the connection vector set of the first subgroup: Object A , Object B , Object CThe connection vector contains: (a) from Location A (2, 2) to Position B The vector of (4, 2); and (b) from Location A (2, 2) to Position C The vector of (2, 5).

[0235] The object tracking image acquisition system can identify another set of connection vectors for the first subgroup: Object A , Object B , Object C The connection vector contains: (a) from Location A (2, 2) to Position B The vector of (4, 2); and (b) from Position B (4, 2) to Bit Place C The vector of (2, 5).

[0236] The object tracking image acquisition system can identify the connection vector set of the second subgroup: Object B, Object C , Object D The connection vector contains: (a) from Position B (4, 2) to Position C The vector of (2, 5); and (b) from Position D (5, 5) to Position C The vector of (2, 5).

[0237] The object tracking image acquisition system can identify the connection vector set of the third subgroup: object A, Object D , Object E The connection vector contains: (a) from Position D (5, 5) to Location A The vector of (2, 2); and (b) from Position D (5, 5) to Position E The vector of (5, 4).

[0238] One or more embodiments include determining a specific set of connection vectors within a set of connection vectors, which is associated with the minimum difference between the connection vector sets of target objects in the captured image of the preceding cross-sectional surface (operation 826). The object tracking image acquisition system compares each set of connection vectors identified at operation 824 with the connection vector set of target objects in the captured image of the preceding cross-sectional surface. For each set of connection vectors identified at operation 824, the object tracking image acquisition system determines (a) the difference between the connection vector set of target objects in the captured image of the preceding cross-sectional surface and (b) the connection vector set identified at operation 824. This calculation is repeated for each set of connection vectors identified at operation 824, thus determining a set of differences. The object tracking image acquisition system identifies the minimum difference among these sets of differences. The object tracking image acquisition system identifies a specific set of connection vectors connecting specific subgroups of objects of interest shown in the segmented image, which is associated with the minimum difference between the connection vector sets of target objects in the captured image of the preceding cross-sectional surface.

[0239] In this embodiment, the difference between the first and second connected vector sets is first determined by identifying vector pairs and comparing them with each other. Each pair of vectors contains (a) a vector from the first connected vector set and (b) a vector from the second connected vector set. Each vector within the first and second connected vector sets is contained in a single pair of vectors. The sum of the absolute values ​​of the differences between each pair of vectors constitutes the difference between the first and second connected vector sets.

[0240] As an example, the set of connection vectors for a group of target objects shown in the captured image of the previous cross-sectional surface contains... Vector A (1, 1) and Vector B (4, 3). The set of connection vectors connecting the first subgroup of objects of interest shown in the segmented image of the current cross-sectional surface contains... Vector C (1, 2) and Vector D (3, 3). The set of connection vectors connecting the second subgroup of objects of interest shown in the segmented image of the current cross-sectional surface contains... Vector E (2, 2) and Vector F (3, 3).

[0241] First, determine the difference between (a) the connection vector of the target object and (b) the connection vector of the first subgroup of objects of interest. The vector pair used to calculate the difference is:

[0242] First pair of vectors: Vector A (1, 1) and Vector C (1, 2); and

[0243] The second pair of vectors: Vector B(4, 3) and Vector D (3, 3).

[0244] For the first pair of vectors, Vector A (1, 1) and Vector C The difference between (1, 2) is calculated as (0, -1). The absolute value of (0, -1) is 1.

[0245] For the second pair of vectors, Vector B (4, 3) and Vector D The difference between (3, 3) is calculated as (1, 0). The absolute value of (1, 0) is 1.

[0246] The sum of the absolute values ​​of the differences between each pair of vectors is calculated as 1 + 1 = 2. Therefore, the difference between the connection vector of (a) the target object and the connection vector of the first subgroup of the object of interest (b) is 2.

[0247] Second, determine the difference between (a) the connection vector of the target object and (b) the connection vector of the second subgroup of the objects of interest. The vector pair used to calculate the difference is:

[0248] First pair of vectors: Vector A (1, 1) and Vector E (2, 2); and

[0249] The second pair of vectors: Vector B (4, 3) and Vector F (3, 3).

[0250] For the first pair of vectors, Vector A (1, 1) and Vector E The difference between (2, 2) is calculated as (-1, -1). The absolute value of (-1, -1) is 1.41.

[0251] For the second pair of vectors, Vector B (4, 3) and Vector F The difference between (3, 3) is calculated as (1, 0). The absolute value of (1, 0) is 1.

[0252] The sum of the absolute values ​​of the differences between each pair of vectors is calculated as 1.41 + 1 = 2.41. Therefore, the difference between the connection vector of (a) the target object and the connection vector of the second subgroup of the object of interest (b) is 2.41.

[0253] Comparing the differences, difference 2 is less than difference 2.41. Therefore, the connection vector set of the first subgroup of the object of interest is associated with the minimum difference of the connection vector set of the target object in the captured image of the previous cross-sectional surface.

[0254] One or more embodiments include identifying a specific subgroup of objects of interest corresponding to a specific set of connection vectors identified at operation 826 as a target object group (operation 828). As described above, the object tracking image acquisition system identifies a specific set of connection vectors associated with the minimum difference of the set of connection vectors of target objects in the captured image of the preceding cross-sectional surface at operation 826. The object tracking image acquisition system identifies a specific subgroup of objects of interest corresponding to the specific set of connection vectors. The specific subgroup of objects of interest is identified as the target object group.

[0255] One or more embodiments include identifying the location of a target group in a segmented image of the current cross-sectional surface (operation 830). As described above at operation 828, a specific subgroup of objects of interest shown in the segmented image of the current cross-sectional surface is identified as a target object group. The location in the segmented image of the current cross-sectional surface corresponding to the target object group is identified. An example of an operation for identifying the location of a target group in a specific image is described above with reference to operation 808.

[0256] One or more embodiments include determining a movement vector (operation 832) from the target group location in a captured image of a previous cross-sectional surface to the target group location in a segmented image of the current cross-sectional surface. Reference above to operation 208 describes an example of an operation for determining a movement vector from one location in an image to another location in the image.

[0257] One or more embodiments include moving the field of view of the microscope according to a motion vector (operation 834). The example of an operation for moving the field of view of the microscope according to a motion vector is described above with reference to operation 210.

[0258] One or more embodiments include obtaining another captured image of the current cross-sectional surface of the specimen (operation 836). Reference above to operation 212 describes an example of an operation for obtaining another captured image of a particular cross-sectional surface after shifting the field of view of the microscope to center on a set of target objects in the particular cross-sectional surface.

[0259] In this embodiment, operation 836 is not performed. Even if the captured image of the current cross-sectional surface obtained at operation 818 is not necessarily centered on the target object, the captured image obtained at operation 818 is accepted as an image of the current cross-sectional surface for 3D rendering of the specimen.

[0260] One or more embodiments include determining whether any additional cross-sectional surface to be observed exists (operation 838). The example of an operation for determining whether any additional cross-sectional surface to be observed is described above with reference to operation 230.

[0261] If an additional cross-sectional surface exists to be observed, one or more embodiments include setting the next cross-sectional surface as the current cross-sectional surface to prepare for obtaining the next captured image (operation 840). Reference to operation 214 above describes an example of an operation for preparing the next cross-sectional surface for observation under a microscope.

[0262] The operation is repeated for the current cross-sectional surface (818 to 836). Therefore, the object tracking image acquisition system traverses each cross-sectional surface of the specimen. After acquiring a captured image of a particular cross-sectional surface, the object tracking image acquisition system moves the microscope's field of view to prepare for acquiring a captured image of the next cross-sectional surface. The object tracking image acquisition system thus tracks the target object group across multiple cross-sectional surfaces of the specimen.

[0263] After observing all cross-sectional surfaces, one or more embodiments include compiling a 3D rendering of the specimen that tracks the target object (Operation 842).

[0264] In this embodiment, operations 814 and 836 are performed. The captured images at operations 814 and 836 are compiled to generate a 3D rendering of the specimen. Therefore, when generating the 3D rendering of the specimen, the captured images at operations 802 and 818 are not compiled.

[0265] In this embodiment, operations 814 and 836 are not performed. Therefore, the captured images at operations 802 and 818 are compiled to generate a 3D rendering of the sample.

[0266] The above reference operation 234 describes an example of an operation for compiling a 3D rendering of a sample that tracks a target object.

[0267] Figures 9A to 9C A set of example operations are shown according to one or more embodiments for obtaining continuous sample images of tracking multiple target objects. Figures 9A to 9C One or more operations shown can be modified, rearranged, or omitted together. Therefore, Figures 9A to 9C The specific order of operations shown should not be construed as limiting the scope of one or more embodiments.

[0268] One or more embodiments include acquiring a captured image of the sample during a first time interval (operation 902). The example of an operation for acquiring a captured image of the sample during a first time interval is described above with reference to operation 302.

[0269] One or more embodiments include identifying a set of target objects in a captured image (operation 904). The above-described example of an operation for identifying target objects in a captured image is referenced to operation 204.

[0270] One or more embodiments include identifying a first set of connection vectors (operation 906) for connecting target objects in a captured image. The foregoing reference to operation 806 describes an example of an operation for identifying a set of connection vectors connecting a particular group of objects.

[0271] One or more embodiments include identifying the location of a target object in a captured image (operation 908). The foregoing reference to operation 808 describes an example of an operation for identifying the location of a group of targets in a particular image.

[0272] One or more embodiments include determining a movement vector (operation 910) from the center of a captured image to the location of a target object in the captured image. Reference operation 208 describes an example of an operation for determining a movement vector from one location in an image to another location in the image.

[0273] One or more embodiments include moving the microscope's field of view relative to the specimen according to a motion vector (operation 912). The example of operation for moving the microscope's field of view according to a motion vector is described above with reference to operation 210. The field of view moves in real time. As the field of view moves, the specimen continues to change over time. It is impossible to obtain another captured image of the specimen in the first time interval because the first time interval has passed and the appearance of the specimen has changed.

[0274] One or more embodiments include obtaining a captured image of the sample at the current time interval (operation 914). The example of an operation for obtaining a captured image of the sample at the current time interval is described above with reference to operation 312.

[0275] One or more embodiments include generating a segmented image based on a captured image (operation 916). The example of an operation for generating a segmented image based on a captured image is described above with reference to operation 218.

[0276] One or more embodiments include identifying a subgroup of objects of interest shown in a segmented image (operation 918). The example of an operation for identifying a subgroup of objects of interest shown in a segmented image was described above with reference to operation 822.

[0277] One or more embodiments include identifying a set of connection vectors that correspond to subgroups of objects of interest in a segmented image (operation 920). The example of an operation for identifying a set of connection vectors that correspond to subgroups of objects of interest in a segmented image is described above with reference to operation 824.

[0278] One or more embodiments include determining a specific set of connection vectors in a set of connection vectors, which is associated with the minimum difference of the set of connection vectors of a target object in a captured image of a previous time interval (operation 922). Reference above to operation 826 describes an example of an operation for determining a specific set of connection vectors associated with the minimum difference of the set of connection vectors of a target object in a previous captured image.

[0279] In one embodiment, the captured image of the immediately preceding time interval is used. A specific set of connection vectors is determined that is associated with the minimum difference of the set of connection vectors of the target objects in the captured image of the immediately preceding time interval. In another embodiment, the captured image of the time interval preceding the immediately preceding time interval is used. This may be necessary due to insufficient processing speed. As an example, images may be captured every 0.1 seconds, but the processing time may be 1 second. In this case, captured images from 10 time intervals prior to the current time interval can be used.

[0280] One or more embodiments include identifying a specific subgroup of objects of interest corresponding to a particular set of connection vectors as target objects (operation 924). The foregoing reference to operation 828 describes an example of an operation for identifying a specific subgroup of objects of interest corresponding to a particular set of connection vectors as target objects.

[0281] One or more embodiments include identifying the location of a target group in a segmented image at the current time interval (operation 926). The above-described operation 830 describes an example of an operation for identifying the location of a target group in a particular image.

[0282] One or more embodiments include determining a movement vector (operation 928) from the target group location in a captured image of a previous time interval to the target group location in a segmented image of the current time interval. Reference above to operation 208 describes an example of an operation for determining a movement vector from one location in an image to another location in the image.

[0283] One or more embodiments include moving the field of view of the microscope according to a motion vector (operation 930). The example of an operation for moving the field of view of the microscope according to a motion vector is described above with reference to operation 210.

[0284] One or more embodiments include determining whether the current time is still within a time period of interest (operation 932). The example of an operation for determining whether the current time is still within a time period of interest was described above with reference to operation 324.

[0285] If the time period of interest has not yet passed, then operation 914 to 930 is repeated for each time interval. Thus, the object tracking image acquisition system acquires captured images of the specimen within the time period of interest. After acquiring a captured image at each time interval, the object tracking image acquisition system moves the microscope's field of view to prepare for acquiring a captured image in the next time interval. The object tracking image acquisition system thus tracks the target object within the time intervals of interest.

[0286] After the time period of interest has passed, one or more embodiments include compiling successive sample images of the target object over time (operation 934). The captured images at operations 902 and 914 are compiled into a set of successive sample images of the target object.

[0287] The above reference operation 326 describes an example of an operation for compiling continuous sample images of a target object that tracks over time.

[0288] 6. Example Implementation

[0289] For clarity, detailed examples are described below. The components and / or operations described below should be understood as specific instances that may not be applicable to some embodiments. Therefore, the components and / or operations described below should not be construed as limiting the scope of any claims.

[0290] Figures 10A to 10N Examples of obtaining continuous sample images for tracking multiple target objects are shown according to one or more embodiments.

[0291] Figure 10A The specimen 1000 is displayed at time T1. Several objects are presented on the surface of the specimen 1000, including object of interest 1052 and object of no interest 1054. A first set of values ​​for the microscope's positioning parameters produces a field of view 1012 relative to the specimen 1000. Therefore, the area of ​​the specimen 1000 within the field of view 1012 can be observed by the microscope, while the area outside the field of view 1012 cannot be observed by the microscope.

[0292] Figure 10B Image 1022, captured at time T1, shows the area of ​​sample 1000 within the field of view 1012. Image 1022 also shows both object of interest 1052 and object of non-interest 1054.

[0293] The user views the captured image 1022. The user enters user input to identify a set of three target objects 1010. The object tracking image acquisition system identifies the connection vector set 1060 connecting the target objects 1010.

[0294] Figure 10CSimilarly, image 1022, captured at time T1 for sample 1000, is also shown. The object tracking image acquisition system also determined the center of target object 1010 at... Position X Place. Position X It can be associated with coordinates (5, 5). Position X The location of the target group is considered to be in the captured image 1022.

[0295] The time interval has passed, and the current time is T2. Figure 10D The specimen 1000 is shown at time T2. The object of interest 1052 and the object of no interest 1054, which were presented on the specimen at time T1, are also presented on the specimen at time T2. However, the objects may be presented in different shapes and / or different positions. The first set of values ​​for the positioning parameters is still used to configure the microscope. The first set of values ​​for the positioning parameters produces a field of view 1014 relative to the specimen 1000. Therefore, the area of ​​the specimen 1000 within the field of view 1014 can be observed through the microscope, while the area outside the field of view 1014 cannot be observed through the microscope.

[0296] Figure 10E Image 1024, captured at time T2, shows the region of sample 1000 within the field of view 1014. Image 1024 also shows both object of interest 1052 and object of non-interest 1054. It should be noted that the group of target objects 1010 is not necessarily located at the center of image 1024.

[0297] Figure 10F The image 1024 captured based on sample 1000 is shown as a segmented image 1034 generated at time T2. Segmented image 1034 indicates a specific segment showing the object of interest 1052. Segmented image 1034 indicates that no object of interest is shown in any region outside the specific segment.

[0298] The object tracking image acquisition system identifies subgroups of objects of interest 1052, where each subgroup has the same number of objects of interest as the target object 1010 (the number is three). The object tracking image acquisition system draws a connection vector for each subgroup of objects of interest 1010.

[0299] Figure 10G Similarly, the segmented image 1034 generated at time T2 based on the captured image 1024 of sample 1000 is shown. Connectivity vector 1062 is drawn for the first subgroup of object of interest 1010.

[0300] The object tracking image acquisition system determines the difference between connection vector 1060 and connection vector 1062. The difference can be, for example, 8.54.

[0301] Figure 10HSimilarly, the segmented image 1034 generated at time T2 based on the captured image 1024 of sample 1000 is shown. Connectivity vector 1064 is drawn for the second subgroup of object of interest 1010.

[0302] The object tracking image acquisition system determines the difference between connection vector 1060 and connection vector 1064. The difference can be, for example, 2.43.

[0303] Figure 10I Similarly, the segmented image 1034 generated at time T2 based on the captured image 1024 of sample 1000 is shown. Connectivity vector 1066 is drawn for the third subgroup of object of interest 1010.

[0304] The object tracking image acquisition system determines the difference between connection vector 1060 and connection vector 1064. The difference can be, for example, 3.89.

[0305] Figure 10J Similarly, the segmented image 1034 generated at time T2 based on the captured image 1024 of sample 1000 is shown. Connectivity vector 1068 is drawn for the fourth subgroup of object of interest 1010.

[0306] The object tracking image acquisition system determines the difference between connection vector 1060 and connection vector 1068. The difference can be, for example, 1.72.

[0307] Comparing the differences, the difference of 1.72 corresponding to connection vector 1068 is the smallest. Therefore, the subgroup of objects of interest corresponding to connection vector 1068 is identified as the group of target objects 1050.

[0308] Figure 10K Similarly, a segmented image 1034 is generated at time T2 based on image 1024 captured from sample 1000. The object tracking image acquisition system determines the center of target object 1050 in... Position Y Place. Position Y It can be associated with coordinates (4, 4.5). Position Y The location of the target group is considered in the segmented image 1034.

[0309] Figure 10L Showing the location of the target group from captured image 1022 ( Position X ) to the target group location in segmented image 1034 ( Position Y The movement vector is 1056. The movement vector can be represented as (-1, 0.5). The movement vector is -1 unit in the x-direction and 1 unit in the y-direction.

[0310] The time interval has elapsed, and the current time is T3. The process is repeated for sample 1000 at time T3. Figure 10M Sample 1000 is shown at time T3. Object of interest 1052 and object of no interest 1054, which were presented on sample 700 at times T1 and T2, are also presented on sample 700 at time T3. However, the objects may be presented in different shapes and / or different positions.

[0311] The second set of positioning parameters is configured for use with the microscope. This second set of positioning parameters moves the field of view according to the movement vector 1056. Specifically, the distance between the previously used field of view 1014 and the current field of view 1016 is equal to the movement vector 1056. Therefore, the area of ​​the sample 1000 within the field of view 1016 can be observed with the microscope, while the area outside the field of view 1016 cannot be observed with the microscope.

[0312] The microscope's field of view moves in real time. As the microscope's field of view moves, the sample 1000 continues to change. Therefore, from time T2 to time T3, the shape and / or position of the target object 1050 has changed. Therefore, even if the field of view has been updated according to the movement vector 1056, the target object 1050 is not necessarily at the center of the current field of view 1016.

[0313] Figure 10N Image 1026, captured at time T3, shows the area of ​​sample 1000 within the field of view 1016. Image 1026 shows both object of interest 1052 and object of non-interest 1054. It should be noted that the target object 1050 is not necessarily located at the center of the captured image 1026.

[0314] Therefore, the process can be repeated for each time interval within the time period of interest. Each captured image (e.g., captured images 1022, 1024, 1026) is compiled to generate an animation showing the changes of the sample over time.

[0315] 7. Hardware Overview

[0316] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute these techniques, or it may contain digital electronic devices, such as one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or network processing units (NPUs), which are persistently programmed to execute these techniques, or it may contain one or more general-purpose hardware processors programmed to execute these techniques according to program instructions in firmware, memory, other storage, or combinations thereof. Such dedicated computing devices may also combine custom hardwired logic, ASICs, FPGAs, or NPUs with custom programming to implement these techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that combines hardwired and / or program logic to implement these techniques.

[0317] For example, Figure 11 This is a block diagram illustrating a computer system 1100 on which embodiments of the present invention may be implemented. The computer system 1100 includes a bus 1102 or other communication mechanism for transmitting information, and a hardware processor 1104 connected to the bus 1102 for processing information. The hardware processor 1104 may be, for example, a general-purpose microprocessor.

[0318] Computer system 1100 also includes main memory 1106, such as random access memory (RAM) or other dynamic storage device, connected to bus 1102, which stores information and instructions to be executed by processor 1104. Main memory 1106 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 1104. When such instructions are stored in a non-transitory storage medium accessible to processor 1104, computer system 1100 presents itself as a dedicated machine customized to perform the operations specified in the instructions.

[0319] The computer system 1100 further includes a read-only memory (ROM) 1108 or other static storage device connected to the bus 1102 for storing static information and instructions for the processor 1104. A storage device 1110, such as a disk or optical disk, is provided and connected to the bus 1102 for storing information and instructions.

[0320] Computer system 1100 can be connected to display 1112, such as a cathode ray tube (CRT), via bus 1102 for displaying information to the computer user. Input device 1114, including alphanumeric keys and other keys, is connected to bus 1102 for transmitting information and command selections to processor 1104. Another type of user input device is cursor controller 1116, such as a mouse, trackball, or cursor arrow keys, which is used to convey directional information and command selections to processor 1104 and control cursor movement on display 1112. This input device typically has two degrees of freedom on two axes (a first axis (e.g., x) and a second axis (e.g., y)), allowing the device to specify a position in a plane.

[0321] Computer system 1100 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, when combined with the computer system, enable or program the computer system 1100 to become a special-purpose machine. According to one embodiment, the techniques herein are executed by computer system 1100 in response to processor 1104 executing one or more sequences of one or more instructions contained in main memory 1106. Such instructions may be read into main memory 1106 from another storage medium, such as storage device 1110. Execution of the instruction sequence contained in main memory 1106 causes processor 1104 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0322] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 1110. Volatile media include dynamic memory, such as main memory 1106. Common forms of storage media include, for example, floppy disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, flash memory-EPROMs, NVRAMs, any other memory chips or cartridges, content-addressable memory (CAM), and tri-state content-addressable memory (TCAM).

[0323] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, and include conductors such as bus 1102. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0324] Various forms of media may be involved when carrying one or more sequences of one or more instructions to processor 1104 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 1100 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 1102. Bus 1102 carries the data to main memory 1106, from which processor 1104 retrieves and executes the instructions. The instructions received in main memory 1106 may optionally be stored on storage device 1110 before or after execution by processor 1104.

[0325] Computer system 1100 also includes a communication interface 1118 coupled to bus 1102. Communication interface 1118 provides bidirectional data communication to network link 1120, which is connected to local network 1122. For example, communication interface 1118 may be an Integrated Services Digital Network (ISDN) card, cable modem, satellite modem, or modem to provide data communication connectivity to a corresponding type of telephone line. As another example, communication interface 1118 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 1118 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0326] Network link 1120 typically provides data communication to other data devices via one or more networks. For example, network link 1120 may provide a connection to host computer 1124 or to data equipment operated by Internet Service Provider (ISP) 1126 via local network 1122. ISP 1126, in turn, provides data communication services via a global packet data communication network now commonly referred to as the "Internet" 1128. Both local network 1122 and Internet 1128 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, as well as signals on network link 1120 and through communication interface 1118 (which carry digital data to and from computer system 1100), are example forms of transmission media.

[0327] Computer system 1100 can send and receive messages, including program code, via a network, network link 1120, and communication interface 1118. In an Internet instance, server 1130 can transmit application request codes via the Internet 1128, ISP 1126, local network 1122, and communication interface 1118.

[0328] The received code may be executed by processor 1104 upon receipt and / or stored in storage device 1110 or other non-volatile memory for later execution.

[0329] 8. Miscellaneous; Extension

[0330] The embodiments relate to a system having one or more means, the means including a hardware processor and configured to perform any of the operations described herein and / or any of the appended claims.

[0331] In an embodiment, the non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause any operation described herein and / or any one of the claims to be performed.

[0332] According to one or more embodiments, any combination of the features and functions described herein may be used. In the foregoing specification, embodiments have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. The unique and exclusive indications of the scope of the invention, and the indications that the applicant intends to define as the scope of the invention, are the literal and equivalent scope of the group of claims published in the specific form of these claims in this application, including any subsequent modifications.

Claims

1. A method for object tracking in a solid sample, comprising: A first image corresponding to the specimen is obtained, and the first image corresponding to the specimen shows the first cross-sectional surface of the specimen; Identify the first position in the first image corresponding to the target object; Thin slices are solidly cut from the surface of the sample; A second image corresponding to the specimen is obtained, wherein the second image is captured by a microscope, and the second image corresponding to the specimen shows a second cross-sectional surface of the specimen; Image segmentation techniques are applied to the second image to obtain segmented images, wherein the segmented images indicate (a) a first set of segments corresponding to objects of interest and (b) a second set of segments that do not correspond to any objects of interest; The specific object of interest among the objects of interest shown in the segmented image that has the highest similarity score to the target object shown in the first image is identified as the target object shown in the segmented image; Identify a second location in the segmented image corresponding to the target object; Determine the movement vector from the first position in the first image to the second position in the segmented image; The field of view of the microscope is moved relative to the specimen according to the movement vector to capture a third image corresponding to the specimen; The method is performed by at least one device containing a hardware processor.

2. The method according to claim 1, further comprising: After moving the field of view of the microscope relative to the specimen according to the movement vector: capturing the third image corresponding to the specimen, the third image showing the second cross-sectional surface of the specimen.

3. The method according to claim 1, further comprising: Obtain the third image corresponding to the specimen, the third image showing the second cross-sectional surface of the specimen; The third image is captured by the microscope after the field of view of the microscope has been moved relative to the sample according to the movement vector; Compile and trace a set of images of the target object, the set of images including the third image but excluding the second image.

4. The method of claim 1, further comprising: After moving the field of view of the microscope relative to the specimen according to the movement vector: capturing the third image corresponding to the specimen, the third image showing the third cross-sectional surface of the specimen.

5. The method of claim 1, further comprising: Obtain the third image corresponding to the specimen, the third image showing the third cross-sectional surface of the specimen; The third image is captured by the microscope after the field of view of the microscope has been moved relative to the sample according to the movement vector; Compile and trace a set of images of the target object, the set of images including the second image and the third image.

6. The method according to claim 1, wherein: The first image corresponding to the sample is shown in the sample at a first time interval; The second image corresponding to the sample shows the sample at a second time interval following the first time interval.

7. The method of claim 1, wherein identifying the specific object of interest among the objects of interest shown in the segmented image that has the highest similarity score with the target object shown in the first image as the target object shown in the segmented image comprises: Determine the location in the first image that is closest to the specific object of interest.

8. The method of claim 1, wherein identifying the specific object of interest among the objects of interest shown in the segmented image that has the highest similarity score with the target object shown in the first image as the target object shown in the segmented image comprises: The first shape of the specific object of interest is determined to be most similar to the second shape of the target object shown in the first image.

9. The method of claim 1, wherein the first image is captured by the microscope.

10. The method of claim 1, wherein the first image is a segmented version of another image captured by the microscope.

11. The method of claim 1, wherein the image segmentation technique includes using an artificial neural network (ANN).

12. A method for object tracking in a solid sample, comprising: A first image corresponding to the specimen is obtained, and the first image corresponding to the specimen shows the first cross-sectional surface of the specimen; Identify a first vector set that connects multiple target objects shown in the first image; Identify the first position in the first image corresponding to the plurality of target objects; Thin slices are solidly cut from the surface of the sample; A second image corresponding to the specimen is obtained, wherein the second image is captured by a microscope, and the second image corresponding to the specimen shows a second cross-sectional surface of the specimen; Image segmentation techniques are applied to the second image to obtain segmented images, wherein the segmented images indicate (a) a first set of segments corresponding to objects of interest and (b) a second set of segments that do not correspond to any objects of interest; Identify subgroups of the objects of interest shown in the segmented image; Identify vector sets corresponding to the subgroups of the objects of interest shown in the segmented image; Determine the specific vector set in the vector set that is associated with the minimum difference from the first vector set; The specific subgroup of the objects of interest connected by the specific vector set is identified as the plurality of target objects; Identify a second location in the segmented image corresponding to the plurality of target objects; Determine the movement vector from the first position in the first image to the second position in the segmented image; The field of view of the microscope is moved relative to the specimen according to the movement vector to capture a third image corresponding to the specimen; The method is performed by at least one device containing a hardware processor.

13. The method according to claim 12, wherein: The first image corresponding to the target object is displayed on the sample at a first time interval; The second image corresponding to the target object is shown in the sample at a second time interval following the first time interval.

14. The method of claim 13, further comprising: After the field of view of the microscope is moved relative to the specimen according to the movement vector, a third image corresponding to the specimen is captured, the third image showing the specimen at a third time interval after the second time interval.

15. The method of claim 13, further comprising: A third image corresponding to the sample is obtained, the third image showing the sample at a third time interval after the second time interval; The third image is captured by the microscope after the field of view of the microscope has been moved relative to the sample according to the movement vector; Compile and trace a set of images of the multiple target objects, the set of images including the second image and the third image.

16. A non-transitory computer-readable medium comprising instructions that, when executed by one or more hardware processors, cause the method according to any one of claims 1 to 15 to be performed.

17. A system for object tracking in a solid specimen, comprising: At least one device including a hardware processor; and The system is configured to perform the method according to any one of claims 1 to 15.

18. A system for tracking an object in a solid specimen, comprising one or more components for performing the method according to any one of claims 1 to 15.

Images