Three-dimensional detection microscope system

Abstract

To provide a three-dimensional microscope system which can perform three-dimensional image detection at a high speed in real time without modification of a normal optical microscope and can perform three-dimensional image feedback that has been conventionally two-dimensional.SOLUTION: A three-dimensional microscope system for performing three-dimensional detection of an observation object comprises: a microscope 2; a plurality of cameras 4 which image an observation object 3 of the microscope 2; and an image processing unit 71 which laminates the plural images of the observation object 3 acquired by the cameras 4 to convert the images into three-dimensional images. A camera adapter 5 is attached to a space between the camera 4 and the microscope 2.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional detection microscope, and more particularly to a three-dimensional detection microscope system capable of three-dimensional detection in real time. [Background technology]

[0002] Conventional microscopes capable of observing minute areas have a shallow focal depth, so they can only obtain two-dimensional images at a certain focal point, and cannot obtain information in the height direction. For example, as shown in FIG. 3(a), when the observation object 3 is a spherical minute object represented by three-dimensional information on the X-axis, Y-axis, and Z-axis, and the needle part 65 of the manipulator 6 is inserted into the center P of the observation object 3 to observe it, the focal plane S1 of the observation object obtained from the microscope is three-dimensional information of the focal contact points S11, S12, and S13, but as shown in FIG. 3(b), the center Q within the focal plane merely represents the center of a two-dimensional image based on two-dimensional information on the Y-axis and Z-axis. Therefore, as shown by "X?" in Fig. 3(a) among the three-dimensional information of the X-axis, Y-axis, and Z-axis of the observation object 3, if the X-axis information in the height direction cannot be obtained, it is not possible to know how much there is a deviation in the height direction of the X-axis, which is the difference between the needle part 65 and the center P of the observation object 3. When observing an observation object 3 with a three-dimensional shape that has such three-dimensional information, there is a problem in that the size in the height direction cannot be known.

[0003] Therefore, to solve this problem, a method for performing three-dimensional observation under a microscope field of view is required. For example, there are two main methods for performing three-dimensional observation using a microscope. One is to observe from two directions and combine the obtained observation images into a three-dimensional image. The other is to observe while changing the focal height from S1 to S5, as shown in Figure 4(a), and combine these obtained multiple observation images into a three-dimensional (stacked image) of a three-dimensional shape like the observed object 3.

[0004] However, this type of focal height change method requires changing the focal height each time a 3D image is acquired, which can cause a lack of real-time performance in applications that require image feedback, such as manipulators. Ultimately, in conventional techniques, real-time image feedback with a normal microscope configuration is only possible with two-dimensional information on a certain focal plane, and information in the height direction cannot be obtained, making it impossible to implement in three dimensions.

[0005] Therefore, methods for acquiring images at multiple focal heights at high speed include a method of changing the phase of illumination light to make it special, as in Patent Document 1, and a method of using a liquid lens as the objective lens to change the focal height at high speed. However, these methods have the problem of requiring the development or modification of a dedicated microscope. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2020-79929 Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention was made to solve these problems, and its purpose is to enable high-speed, real-time, three-dimensional image detection without modifying an ordinary optical microscope, and to enable three-dimensional image feedback, which was previously only two-dimensional. [Means for solving the problem]

[0008] To this end, the present invention comprises a microscope and a plurality of cameras for photographing the object observed by the microscope; an image processing unit that stacks and three-dimensionally creates multiple images of the observed object acquired via the multiple cameras; A three-dimensional microscope system for three-dimensional detection of an observed object, comprising: each of the cameras is connected to at least the lens barrel of the microscope; a camera adapter is disposed between each of the cameras and the lens barrel, and a distance between the microscope and the camera adapter is set to be different for each of the plurality of cameras; the image processing unit simultaneously acquires a plurality of images of the object of observation at different focal heights by the camera, and performs layered image processing; The generation of the three-dimensional detection of the stacked images of the observed object is performed with the aid of an artificial intelligence program; a manipulator for finely moving the object under observation in the microscope, and a control unit for driving and controlling the manipulator based on data from the three-dimensional detection, the manipulators are a holding manipulator and an injection manipulator, each having a function of receiving a three-dimensional detection data signal from the control unit and finely moving the object to be observed in three orthogonal axes, and one holding manipulator is provided on the left side as one faces the microscope, and one injection manipulator is provided on the right side as one faces the microscope; The three-dimensional microscope system is characterized in that the tip of the holding manipulator is provided with a holding manipulator needle part for holding the observation object, and the tip of the injection manipulator is provided with an injection manipulator needle part for insertion into the observation object.

[0009] This configuration allows multiple cameras to be attached to the microscope at different focal heights, and the images obtained can be processed simultaneously to obtain 3D images at high speed. Furthermore, the control device can generate 3D images using fewer layered images using artificial intelligence. [Effects of the Invention]

[0010] According to the present invention, it is possible to realize a three-dimensional detection microscope system that enables high-speed, real-time three-dimensional image detection without modifying an ordinary optical microscope, and that can perform three-dimensional image feedback, which was previously only two-dimensional. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1(a) is a schematic diagram of an embodiment of the present invention, and FIG. 1(b) is a diagram showing the concept of an observed object and a three-dimensional stacked image. [Figure 2] (a) is a diagram showing the case where there is no gap between the microscope of the present invention and the camera adapter and the focusing state, and (b) is a diagram showing the case where there is gap between the microscope of the present invention and the camera adapter and the focusing state. [Figure 3] FIG. 1(a) is a diagram showing the focal position of an object observed in the prior art, and FIG. 1(b) is a diagram showing the inside of the focal plane. [Figure 4] FIG. 1A is a diagram showing a plurality of stacked images of an observed object according to the prior art, and FIG. 1B is a conceptual diagram of an observed object obtained by synthesizing the three-dimensional stacked images. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, one embodiment of a three-dimensional detection microscope system of the present invention will be described with reference to the accompanying drawings 1 and 2. Note that this embodiment is one embodiment of the present invention, and is not to be construed as being limited thereto in any way, and design changes are possible within the scope of the present invention.

[0013] As shown in Fig. 1, this three-dimensional detection microscope system 1 is a microscope system that performs three-dimensional detection of an observation object 3 used in a manipulation system for micromanipulation applications such as cell manipulation, microfabrication, and micro-object pickup, and includes a microscope 2, a plurality of microscope cameras (hereinafter simply referred to as cameras) 4 that capture images of the observation object 3 of the microscope 2, an image processing unit 71 (controller 7) that stacks and three-dimensionalizes the multiple images of the observation object 3 acquired through the cameras 4, and the camera 4 is attached to the microscope 2 via a camera adapter 5. Note that in Fig. 1, the camera 43 (4) is shown separately from the microscope 2 and not attached to it in order to easily explain this embodiment; in reality, as shown in Fig. 2, the camera 4 is attached to the microscope 2 (lens barrel 26) via a camera adapter 5.

[0014] The microscope 2 used in this three-dimensional detection microscope system 1 is a general optical microscope having binocular eyepieces for observing with the naked eye an observation object 3, which is a minute target sample. A high-magnification image of the observation object 3 photographed by such a microscope 2 is input as digital data via a camera 4 to an image processing unit 71 of a control device 7, where image processing for three-dimensional detection is performed.

[0015] As shown in Figure 1, the microscope 2 is fixed in place by a mirror column 22 attached to a mirror leg 21, and the optical axis of the light source 23 is vertical. An object 3 to be observed is placed on a stage 24. An objective lens 25 is used to change the magnification and observe the object 3 on the optical axis.

[0016] The camera 4 is a digital camera for a microscope that takes an image observed through the microscope 2 and generates image data representing the observed image, and the microscope 2 and the image processing unit 71 of the control device 7 are connected by a cable 27, allowing electrical signals to be sent and received between them, enabling the camera 4 to take a photograph. The image observed through the microscope 2 can be confirmed on a monitor (not shown) of the control device 7.

[0017] In this embodiment, as shown in FIG. 1, manipulators 6 for finely moving the observation object 3 are provided on both the left and right sides of the microscope. The manipulator 6 has the function of receiving three-dimensional detection data (X-axis, Y-axis, Z-axis) signals from the control unit 73 and finely moving the microscopic object, which is the observation object 3 of the microscope 2, along three orthogonal axes (X-axis, Y-axis, Z-axis). The manipulator 6 is equipped with a holding manipulator 61 on the left side as viewed from the microscope 2, and an injection manipulator 64 on the left side. The tip of the holding manipulator 61 is equipped with a holding manipulator needle part 62, and the tip of the left injection manipulator 64 is equipped with an injection manipulator needle part 65.

[0018] The holding manipulator 61 is a manipulator configured with three orthogonal axes (X-axis, Y-axis, and Z-axis), and is driven and controlled by a holding manipulator drive device 63 to move within a three-dimensional space as a moving area, and holds a microscopic object or the like that is the observation object 3 on the stage 24. The injection manipulator 64 is also a manipulator configured with three orthogonal axes (X-axis, Y-axis, and Z-axis), and is driven and controlled by an injection manipulator drive device 66 to move within a three-dimensional space that includes the microscopic object or the like that is the observation object 3 on the stage 24 as a moving area, and for example, the insertion position for inserting the injection manipulator needle part 65 into the observation object 3 is controlled.

[0019] In this three-dimensional detection microscope system 1 configured as above, the microscope 2 and the camera adapter 5 are mounted so that the distance between each of the multiple cameras 4 is different, and multiple images of the object of observation 3 at different focal heights are simultaneously acquired by the cameras 2 and processed. The principle of three-dimensional detection will be explained using Figures 2(a) and (b). First, as shown in Fig. 2(a), the camera adapter 5 is mounted between the camera 4 of this embodiment and the lens of the lens barrel 26 of the microscope 2, and Fig. 2(a) shows a state in which there is no gap (no interval) between the microscope 2 and the camera adapter 5. In this case, when the needle 62 of the manipulator 6 is observed through the microscope 2, it can be seen that the focal plane is aligned with the middle part of the needle 62 as shown in the ellipse in Fig. 2(a), and the tip of the needle 62 outside the focal plane is out of focus and cannot be seen clearly.

[0020] Next, Figure 2(b) shows a state where the distance (gap) "t" between the microscope 2 and the camera adapter 5 is 1 mm. In this case, when the needle 62 of the manipulator 6 is observed with the microscope 2, the focal plane is as shown within the ellipse in the figure, and the focus is on the tip of the needle 62. In the example of this embodiment shown in Figure 2, when the distance (gap) "t" is changed from zero to 1 mm, a change in the height of the focal point to 50 µm is observed. That is, the focal height can be changed by providing a distance (gap) between the microscope 2 and the camera adapter 5. In other words, to observe multiple focal planes with different focal heights, it is sufficient to provide multiple different distances (gaps) "t" between the microscope 2 and the camera adapter 5.

[0021] In this way, in order to be able to observe images at a plurality of focal planes with different focal heights, the distance (gap) "t" between the microscope 2 and the camera adapter 5 is set to be different for each of the plurality of different cameras 4. In this embodiment, based on this principle, as shown in FIG. 1(a), the distance (gap) between the microscope 2 and the camera adapter 5 is shifted and set differently for each of the three cameras 41, 42, 43. As a result, as shown in FIG. 1(b), three-dimensional information on the X-axis, Y-axis, and Z-axis of the focal planes S1, S3, and S5 of the object of observation 3, which have different focal heights, is detected from each camera and synthesized to generate a three-dimensional image (stacked image) of the object of observation. The images from these three cameras 41, 42, and 43, each with a different focal plane, and the combined image of the object to be observed (stacked image) are displayed on a monitor (not shown) of the control device 7 so that they can be confirmed. In this embodiment, the number of cameras 4 is described as three, but for example, if the focal planes are S1 to S5 as shown in Figure 4(a), five cameras 4 may be provided, and therefore the number of cameras 4 can be changed depending on the detailed conditions of the object 3 to be observed.

[0022] Next, in this embodiment, an artificial intelligence program is provided that is assisted by artificial intelligence to generate a three-dimensional detection in which multiple images of the observed object 3 are layered. Specifically, the three cameras 41, 42, and 43 are set to different focal heights, and microscope images (focal plane images) are acquired at the same time, and as a means for speeding up these and enabling layered three-dimensional detection in real time, an AI (artificial intelligence) 72 program loaded into a memory (not shown) of the control device 7 is combined with the image processing unit 71.

[0023] The AI ​​(artificial intelligence) 72 program uses a large number of learning three-dimensional image data of the observed object 3 at different focal heights to learn, for example, the relationship between the image of the observed object 3 and the focal height, and based on the learning results, calculates and predicts the degree of certainty that the three images with different three-dimensional data received from the three cameras 41, 42, and 43 are the three-dimensional data of the observed object 3. Then, the certainty calculation result including the predicted certainty is notified to the control unit 73 of the control device 7 as a control signal for the manipulator 6. When there are multiple selected certainty calculation results, the image processing unit 71 selects and determines the certainty calculation result that indicates the highest certainty from the multiple certainty calculation results.

[0024] In addition, the AI ​​(artificial intelligence) 72 program may be machine learning, such as supervised learning of neural networks (deep learning), supervised learning using pattern recognition models (support vector machines), or supervised learning using probability models.

[0025] In terms of hardware, it can be configured with circuit blocks, memory, and other LSIs, and images of the observed object 3 at different focal heights are sent from the three cameras 41, 42, and 43 to the image processing unit 71 of the control device 7, and for the multiple image results received from the image processing unit 71 in the AI ​​(artificial intelligence) 72, a confidence level corresponding to the image information learned by the program of the AI ​​(artificial intelligence) 72 is calculated in association with the multiple images received by the image processing unit 71, and a three-dimensional stereoscopic image of the observed object 3 synthesized in three dimensions is output based on the prediction result.

[0026] In this embodiment, the three-dimensional (X-axis, Y-axis, Z-axis) data of the stacked images from the image processing unit 71 is processed by a computer that can learn and determine it using a program of an AI (artificial intelligence) 72. This AI (artificial intelligence) 72 can accurately learn and detect the three-dimensional (X-axis, Y-axis, Z-axis) data of the observed object 3 obtained through the image of the camera 4, including the so-called height distance, position, and shape of the focal point.

[0027] As a result, instead of the conventional method of acquiring images by changing the focus one by one with one camera and then simply synthesizing these images one by one, it is possible to generate a more complete three-dimensional image of the object of observation 3 in instantaneous real time from a small number of images from the three cameras 41, 42, 43 with predictive support from the AI ​​(artificial intelligence) 72 program from stacked images of the object of observation 3 with different focuses that are captured and acquired by the three cameras 41, 42, 43.

[0028] In this way, this three-dimensional detection microscope system can generate three-dimensional images in real time even with a small number of stacked images by having AI (artificial intelligence) 72 predict and select the confidence calculation results, and can realize a three-dimensional detection microscope system 1 in combination with AI (artificial intelligence) 72.

[0029] Therefore, in the conventional technology of changing the focal height, in order to acquire one three-dimensional image, it was necessary to change the focal height one by one and create different focal planes to complete the stacked images, which made image processing slow and lacked real-time capabilities. However, the present three-dimensional detection microscope system 1 can solve this problem of real-time image processing.

[0030] Furthermore, in this three-dimensional detection microscope system, the deviation in the height direction of the X axis between the needle parts 62, 65 of the manipulator 6 and the object of observation 3 and the magnitude of the deviation are analyzed, and a three-dimensional image reflecting this deviation is generated in real time, and feedback control signals are sequentially transmitted to the holding manipulator drive device 63 and the injection manipulator drive device 66 so that the object of observation 3 can be observed accurately.

[0031] In this way, feedback control based on an image of a three-dimensional focal plane is performed in real time, rather than feedback control based on an image of a two-dimensional focal plane, making it possible to accurately and efficiently operate a manipulator in three dimensions, such as cell manipulation or microfabrication of the observation object 3. [Industrial Applicability]

[0032] The three-dimensional detection microscope system of the present invention can be used in micromanipulators in cutting-edge fields such as the medical field and the biological field that use three-dimensional detection microscopes. [Explanation of symbols]

[0033] 1. Three-dimensional detection microscope system 2. Microscope 21 Mirror Leg 22 Mirror pillar 23 Light source 24 stages 25 objective lenses 26 Telescope tube 27 Cable 3 Observation object (microscopic object) 4 Camera (microscope camera) 5 Camera Adapter 6 Manipulator 61 Holding manipulator 62 Holding manipulator needle 63 Holding manipulator drive unit 64 Injection Manipulator 65 Injection manipulator needle 66 Injection manipulator drive unit 7 Control Device 71 Image processing section 72 AI (artificial intelligence) 73 Control Unit

Claims

[Claim 1] A microscope and a plurality of cameras for photographing the object observed by the microscope; an image processing unit that stacks and three-dimensionally creates multiple images of the observed object acquired via the multiple cameras; A three-dimensional microscope system for three-dimensional detection of an observed object, comprising: each of the cameras is connected to at least the lens barrel of the microscope; a camera adapter is disposed between each of the cameras and the lens barrel, and a distance between the microscope and the camera adapter is set to be different for each of the plurality of cameras; the image processing unit simultaneously acquires a plurality of images of the object of observation at different focal heights by the camera, and performs layered image processing; The generation of the three-dimensional detection of the stacked images of the observed object is performed with the aid of an artificial intelligence program; a manipulator for finely moving the object under observation in the microscope, and a control unit for driving and controlling the manipulator based on data from the three-dimensional detection, the manipulators are a holding manipulator and an injection manipulator, each having a function of receiving a three-dimensional detection data signal from the control unit and finely moving the object to be observed in three orthogonal axes, and one holding manipulator is provided on the left side as one faces the microscope, and one injection manipulator is provided on the right side as one faces the microscope; A three-dimensional microscope system characterized in that the tip of the holding manipulator is provided with a holding manipulator needle part for holding the observation object, and the tip of the injection manipulator is provided with an injection manipulator needle part for insertion into the observation object.

Images