Manipulation system and three-dimensional position display method
The manipulation system uses a variable focus lens and imaging to present the three-dimensional positions of cells and manipulators in virtual space, addressing the limitation of viewing from a single direction, enhancing manipulation precision and convenience.
Patent Information
- Application Number
- JP2020166085
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-09-30
AI Technical Summary
Existing manipulation systems for cells using multiple microscopes can only confirm the three-dimensional arrangement relationship from a specific direction, limiting the ability to view from any viewpoint.
A manipulation system and method that utilizes a variable focus lens to change the refractive power and working distance of the optical path, combined with imaging and control devices to capture multiple images, identify the three-dimensional positions of the sample and manipulator, and map their relationship in virtual space for real-time three-dimensional display.
Enables the presentation of three-dimensional positions of the sample and manipulator from any viewpoint, improving the convenience and precision of cell manipulation by allowing real-time, stereoscopic viewing of their arrangement.
Smart Images

Figure 0007584123000006 
Figure 0007584123000007 
Figure 0007584123000008
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a manipulation system. [Background technology]
[0002] There is a known manipulation system for manipulating cells using a manipulator. Because cells are very small, the positions of the cells and the manipulator are observed under a microscope while the cells are manipulated. For example, a system has been proposed that allows the three-dimensional positional relationship between the cells and the manipulator to be grasped by arranging multiple microscopes to observe the cells from multiple directions. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-122891 A Summary of the Invention [Problem to be solved by the invention]
[0004] When multiple microscopes are combined, the relative positions of the microscopes can only be confirmed when viewed from a specific direction. It is preferable to be able to confirm the three-dimensional relative positions of the microscopes from any viewpoint.
[0005] The present disclosure has been made in view of the above problems, and one of its exemplary objectives is to provide a technique for indicating the three-dimensional positions of a sample and a manipulator. [Means for solving the problem]
[0006] A manipulation system according to one aspect of the present disclosure includes a stage for supporting a sample, a manipulator for manipulating the sample on the stage, a manipulator driving mechanism for moving the manipulator, an objective lens for observing the sample and the manipulator, a variable focus lens into which observation light that has passed through the objective lens is incident, an imaging device that captures the observation light that has passed through the variable focus lens to generate an image, a lens driving mechanism that changes the refractive power of the variable focus lens to change a working distance from the tip of the objective lens to a focal position of the observation light, a control device that identifies a three-dimensional position of the sample based on a plurality of captured images captured while changing the working distance, identifies a three-dimensional position of the manipulator based on operation of the manipulator driving mechanism, and maps the three-dimensional positional relationship of the sample and the manipulator into a virtual space, and a display device that displays the three-dimensional positional relationship of the sample and the manipulator mapped into the virtual space.
[0007] Another aspect of the present disclosure is a three-dimensional position presentation method, which includes the steps of: changing the refractive power of a variable-focus lens into which observation light passing through an objective lens for observing a sample and a manipulator on a stage is incident, thereby changing a working distance from the tip of the objective lens to a focal position of the observation light, capturing the observation light with an imaging device while changing the working distance to generate a plurality of captured images, identifying a three-dimensional position of the sample based on the plurality of captured images, identifying a three-dimensional position of the manipulator based on an operation of the manipulator, mapping a three-dimensional positional relationship between the sample and the manipulator in a virtual space, and displaying the three-dimensional positional relationship between the sample and the manipulator mapped in the virtual space.
[0008] Yet another aspect of the present disclosure is a program that causes a computer to realize a function of controlling a lens drive mechanism for changing the refractive power of a variable focus lens onto which observation light passing through an objective lens for observing a sample and a manipulator on a stage is incident, thereby changing a working distance from the tip of the objective lens to a focal position of the observation light, a function of acquiring a plurality of captured images by capturing the observation light with an imaging device while changing the working distance, a function of identifying a three-dimensional position of the sample based on the plurality of captured images, a function of identifying a three-dimensional position of the manipulator based on the operation of the manipulator, and a function of mapping a three-dimensional arrangement relationship between the sample and the manipulator in a virtual space.
[0009] Any combination of the above components, or mutual substitution of the components or expressions of the present disclosure between methods, systems, etc., are also valid aspects of the present disclosure. Effect of the Invention
[0010] According to the present disclosure, the three-dimensional positions of the sample and the manipulator can be presented. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a manipulation system according to an embodiment. [Diagram 2] FIG. 13 is a diagram illustrating a working distance taking into account the influence of the refractive index. [Diagram 3] FIG. 13 is a diagram showing a schematic method for identifying the position coordinates of a sample in the z direction. [Figure 4] 11 is a graph showing the correlation between the average edge strength approximated by a Gaussian distribution and the working distance. [Diagram 5] 10A and 10B are diagrams illustrating a measurement pattern of a manipulator when deriving coordinate transformation parameters. [Figure 6] 6(a) to 6(j) are diagrams showing examples of captured images within the field of view of the objective lens and display examples of the three-dimensional display device. [Figure 7]4 is a flowchart showing a three-dimensional position presenting method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] First, an overview of the present disclosure will be described. The present disclosure relates to a manipulation system. The manipulation system includes an optical microscope for observing a sample such as a cell, and a manipulator for manipulating the sample. A user manipulates the sample by moving the manipulator while observing the sample and the manipulator with the optical microscope. Since the positions of the sample and the manipulator change in three dimensions, it is difficult to properly manipulate the sample unless the three-dimensional arrangement of the sample and the manipulator is known. In the present disclosure, the three-dimensional positions of the sample and the manipulator are identified in real time, and the three-dimensional arrangement of the sample and the manipulator is presented to the user by mapping both of them in a virtual space. The three-dimensional arrangement of the sample and the manipulator is provided to the user through, for example, a three-dimensional display device capable of stereoscopic viewing. According to the present disclosure, by displaying the three-dimensional positions of the sample and the manipulator three-dimensionally, the user can confirm the arrangement of the sample and the manipulator from any viewpoint, thereby improving the convenience of the manipulation system.
[0013] 1 is a diagram showing a schematic configuration of a manipulation system 10 according to an embodiment. The manipulation system 10 has an inverted microscope configuration. The manipulation system 10 includes a stage 12, an illumination device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator driving mechanism 26, a lens driving mechanism 28, a control device 30, a three-dimensional display device 32, a two-dimensional display device 34, and an input device 36.
[0014] 1, a first coordinate system is set with the optical axis A of the objective lens 20 on the stage 12 as a reference. The direction in which the optical axis A of the objective lens 20 on the stage 12 extends is the z direction, and the directions perpendicular to the optical axis A are the x direction and the y direction. In the example shown, the direction in which the optical axis A of the objective lens 20 on the stage 12 extends coincides with the direction perpendicular to the support surface 12a of the stage 12. Note that the direction in which the optical axis A of the objective lens 20 on the stage 12 extends may deviate from the direction perpendicular to the support surface 12a of the stage 12.
[0015] The stage 12 has a support surface 12a for supporting the sample 40 horizontally, and an opening 12b for passing the observation light 42 from the sample 40. The sample 40 to be manipulated is not particularly limited, but the sample may be cells of a human or animal. The sample 40 is contained in a sample dish 46 made of a transparent material such as resin or glass, and the sample dish 46 is placed on the stage 12. The sample 40 is suspended in a liquid 48 such as water contained in the sample dish 46.
[0016] The illumination device 14 is provided above the stage 12 and illuminates the sample 40 on the stage 12. The illumination device 14 projects illumination light 44, such as white light, toward the sample 40. The illumination device 14 is configured to provide transmitted illumination. The illumination device 14 may be capable of projecting illumination light 44 of visible light of a specific wavelength selected for fluorescence observation or the like. The illumination device 14 projects illumination light 44 such that the illuminance distribution on the stage 12 is uniform, for example.
[0017] The manipulator 16 is provided on the stage 12 and is used to manipulate the sample 40. In the illustrated example, the manipulator 16 includes a holding pipette 16a and an injection pipette 16b. For example, a cell is fixed using the holding pipette 16a, and cell manipulation such as gene introduction into the cell is performed using the injection pipette 16b. In the illustrated example, two manipulators are provided, but the number of manipulators may be one or three or more.
[0018] The folding mirror 18 is provided directly below the opening 12b of the stage 12. The folding mirror 18 is disposed so as to reflect the observation light 42 from the sample 40 toward the objective lens 20. In the illustrated example, the optical axis A of the objective lens 20 is folded back by the folding mirror 18, but the objective lens 20 may be disposed on an optical axis extending in the z direction without providing the folding mirror 18.
[0019] The objective lens 20 is disposed at a position where the observation light 42 from the folding mirror 18 is incident. The objective lens 20 is disposed at a position away from the folding mirror 18 in the +x direction. It is desirable that the objective lens 20 has a relatively long working distance (WD). The specifications of the objective lens 20, such as the magnification and working distance, are not particularly limited, but for example, an ultra-long working distance type objective lens having a working distance of 20 mm to 40 mm at a magnification of 10 times to 50 times can be used.
[0020] The variable-focus lens 22 is disposed at a position where the observation light 42 that has passed through the objective lens 20 is incident. The variable-focus lens 22 is disposed between the objective lens 20 and the imaging device 24, for example, adjacent to or close to the objective lens 20. The variable-focus lens 22 is configured so that its refractive power is variable within a predetermined range. The variable-focus lens 22 may be a convex lens having only positive refractive power, or a concave lens having only negative refractive power, or may be configured so that its refractive power can be switched between positive and negative.
[0021] The variable-focus lens 22 is, for example, a liquid lens, and is configured so that the focal length can be changed by deforming a flexible transparent film that seals the liquid lens. The shape of the transparent film is controlled by changing the pressure applied to the transparent film. For example, the focal length of the variable-focus lens 22 can be electrically controlled by using an electromagnetic actuator or a piezoelectric element. The variable-focus lens 22 is configured so that the effective working distance of the combination of the objective lens 20 and the variable-focus lens 22 can be changed within a range of about 2 mm.
[0022] The imaging device 24 captures the observation light 42 that has passed through the variable focus lens 22 to generate a captured image. The imaging device 24 has an imaging lens 24a and an imaging element 24b. The imaging lens 24a focuses the observation light 42 on the imaging element 24b. The imaging element 24b is an image sensor such as a CMOS sensor, and is capable of generating captured images at a high frame rate. The frame rate of the imaging device 24 is not particularly limited, but is preferably 100 frames per second or more, and more preferably 500 frames per second or more.
[0023] The objective lens 20, the variable focus lens 22, and the image pickup device 24 are arranged along an optical axis A extending in the x direction, and are fixed, for example, to a lens barrel extending in the x direction. Note that an additional folding mirror (not shown) may be provided between the variable focus lens 22 and the image pickup device 24, and the optical axis A may be further folded.
[0024] The manipulator driving mechanism 26 moves the manipulator 16 and varies the three-dimensional position of the manipulator 16. In the illustrated example, the manipulator driving mechanism 26 includes a first driving mechanism 26a and a second driving mechanism 26b. The first driving mechanism 26a is configured to move the holding pipette 16a and vary the three-dimensional position of the holding pipette 16a. The second driving mechanism 26b is configured to move the injection pipette 16b and vary the three-dimensional position of the injection pipette 16b. The three-dimensional positions of the holding pipette 16a and the manipulator 16 can be controlled independently of each other.
[0025] The lens driving mechanism 28 drives the variable-focus lens 22 to change the refractive power of the variable-focus lens 22. The lens driving mechanism 28 changes the refractive power of the variable-focus lens 22 to change the effective working distance of the combination of the objective lens 20 and the variable-focus lens 22. Here, the effective working distance is the distance from the tip of the objective lens 20 to the focal position of the observation light 42, and is the distance from the tip of the objective lens 20 to the focal plane on which the image captured by the imaging device 24 is focused.
[0026] The control device 30 controls the overall operation of the manipulation system 10. In terms of hardware, the control device 30 can be realized by elements and mechanical devices such as a computer CPU and memory, and in terms of software, the control device 30 is realized by a computer program, etc. The control device 30 is configured, for example, by a general-purpose personal computer.
[0027] The three-dimensional display device 32 stereoscopically displays the three-dimensional positions of the sample 40 and the manipulator 16. The three-dimensional display device 32 is, for example, a hologram display such as Looking Glass, and is a display device that allows stereoscopic viewing without using 3D glasses or the like. The three-dimensional display device 32 displays computer graphic (CG) images that mimic the sample 40 and the manipulator 16. The three-dimensional display device 32 may be a head-mounted virtual reality (VR) display device.
[0028] The two-dimensional display device 34 is a liquid crystal display or the like, and displays in real time images captured by the imaging device 24. The two-dimensional display device 34 may display the three-dimensional positions of the sample 40 and the manipulator 16, or may display a rendering image generated by perspectively projecting the sample 40 and the manipulator 16 mapped in a virtual space onto an arbitrary observation surface.
[0029] The input device 36 is a device for inputting data to the control device 30 and operating the manipulator 16. A mouse, a keyboard, or the like can be used as an input device for the control device 30. A joystick or the like can be used as an operating device for the manipulator 16. By using the input device 36, the tip position of the manipulator 16 can be moved on the order of micrometers, and the sample 40 can be precisely operated.
[0030] A method for determining the three-dimensional positions of the sample 40 and the manipulator 16 in real time will now be described.
[0031] The control device 30 specifies the three-dimensional position of the sample 40 included in the captured image based on the captured image captured by the imaging device 24. The control device 30 specifies the three-dimensional position of the sample 40 using a first coordinate system based on the optical axis A of the objective lens 20. The control device 30 specifies the sample 40 included in the captured image using an image recognition technique, and specifies the position coordinates of the sample 40 in the x and y directions from the center position of the sample 40 in the captured image. The control device 30 specifies the position coordinate of the sample 40 in the z direction based on the working distance when the sample 40 is imaged. For example, the control device 30 sets the tip of the objective lens 20 as the origin (z=0) in the z direction, and sets the working distance from the objective lens 20 to the sample 40 as the position coordinate of the sample 40 in the z direction.
[0032] The working distance WD can be calculated based on the focal length f1 of the objective lens 20, the focal length f2 of the variable focus lens 22, the distance d between the objective lens 20 and the variable focus lens 22, and the refractive index distribution of the optical path from the objective lens 20 to the sample 40. The composite focal length f0 of the combination of the objective lens 20 and the variable focus lens 22 is expressed as f0=f1(f2-d) / (f1+f2-d). The composite focal length f0 corresponds to the working distance when the optical path from the objective lens 20 to the sample 40 is air and has a refractive index of approximately 1. In reality, the sample dish 46 and the liquid 48 are present in the optical path from the objective lens 20 to the sample 40. Therefore, the actual working distance WD deviates from the composite focal length f0 due to the influence of the refractive index of the sample dish 46 and the liquid 48.
[0033] FIG. 2 is a diagram showing a schematic diagram of the working distance WD considering the influence of the refractive index. As shown in the figure, the presence of the sample dish 46 and the liquid 48 in the optical path from the objective lens 20 to the sample 40 refracts the observation light 42 incident on the objective lens 20. As a result, the actual working distance WD is longer than the composite focal length f0 of the objective lens 20 and the variable focus lens 22 in the absence of the sample dish 46 and the liquid 48. The actual working distance WD can be expressed by the following formula (1) using the composite focal length f0, the refractive index n0 of air, the refractive index n1 of the sample dish 46, the refractive index n2 of the liquid 48, the distance a from the objective lens 20 to the sample dish 46, the thickness b of the sample dish 46, and the effective radius r when the objective lens 20 is assumed to be an ideal plano-convex lens. Formula (1) can be derived based on the geometrical relationship based on Snell's law.
number
[0034] As an example of the actual working distance WD, when the composite focal length f0 is 21.059 mm, the refractive index n1 of the polystyrene (PS) sample dish 46 is 1.592, the refractive index n2 of the pure water liquid 48 is 1.33, the distance a is 19.135 mm, the thickness b is 1.0 mm, and the effective radius r is 5.0 mm, the working distance WD is 21.894 mm. In this case, the difference between the working distance WD and the composite focal length f0 is 0.835 mm, which is very large compared to the depth of field of the objective lens 20 (about 0.03 mm) and the size of the cell (about 0.1 mm). Therefore, by correcting the working distance WD considering the influence of the refractive index distribution from the objective lens 20 to the sample 40, the position coordinate of the sample 40 in the z direction can be accurately identified.
[0035] The control device 30 controls the lens drive mechanism 28 to change the working distance WD. The control device 30 identifies the z-direction position coordinates of the sample 40 based on a plurality of captured images captured by changing the working distance WD. Specifically, the average edge intensity F of the sample 40 included in each of the plurality of captured images is calculated, and the z-direction position coordinates of the sample 40 are identified based on the maximum value when the correlation between the working distance WD and the average edge intensity F is approximated by a Gaussian distribution. The average edge intensity F is obtained by calculating the edge intensity f(x, y) of each pixel of the captured image and averaging the edge intensities f(x, y) of all the pixels in the region where the sample 40 is included. Assuming the luminance value of each pixel is I(x, y), the edge intensity is f(x, y) = {(I(x + 1, y) - I(x, y)) 2 + (I(x, y + 1) - I(x, y)) 2} 1 / 2 and is represented as such. The average edge intensity F indicates the contrast of the sample 40 included in the captured image, and the larger the average edge intensity F, the more in-focus and high-contrast the sample 40 is captured.
[0036] FIG. 3 is a diagram schematically showing a method for identifying the z-direction position coordinates of the sample 40. In FIG. 3, the working distance WD of the objective lens 20 is set to a first distance z1, a second distance z2, and a third distance z3 (z1 < z2 < z3), and captured images 52a, 52b, and 52c of the focal planes 50a, 50b, and 50c located at each distance z1 to z3 are schematically shown. In the illustrated example, the central coordinate z of the sample 40 is located between the first distance z1 and the second distance z2, showing a state where the sample 40 intersects the second focal plane 50b. Since the second focal plane 50b intersects the sample 40, the second captured image 52b of the second focal plane 50b includes the sample 40 in an in-focus state. On the other hand, the first captured image 52a of the first focal plane 50a away from the sample 40 includes the sample 40 in a slightly out-of-focus state. Also, the third captured image 52c of the third focal plane 50c further away from the sample 40 includes the sample 40 in an out-of-focus state. The interval Δz between the first distance z1, the second distance z2, and the third distance z3 is set to a value about 1 to 2 times the size of the sample 40. For example, if the size of the sample 40 is about 100 μm, then Δz = about 100 μm to 200 μm.
[0037] The control device 30 acquires a plurality of captured images 52a to 52c captured while changing the operating distance WD, and calculates the average edge intensities F1, F2, and F3 of the regions 54a to 54c in which the sample 40 is included in each of the captured images 52a to 52c. In the example shown in FIG. 3, the magnitude relationship of the average edge intensities of the captured images 52a to 52c is F3 < F1 < F2. The control device 30 approximates the correlation between the calculated average edge intensities F1, F2, and F3 and the distances z1, z2, and z3 to the respective focal planes 50a to 50c with a Gaussian distribution.
[0038] FIG. 4 is a graph showing the relationship between the average edge intensities F1, F2, and F3 approximated by a Gaussian distribution and the operating distance WD. As shown in the figure, if the magnitude relationship of the average edge intensities is F1 < F2 and F3 < F2, the average edge intensities F1 to F3 are approximated by a Gaussian distribution to obtain a maximum value F max can be specified. The maximum value F max corresponds to the position where the focus of the sample 40 is the best, so the operating distance z0 corresponding to the maximum value F max can be regarded as the z coordinate of the center of the sample 40. Note that the operating distance z0 corresponding to the maximum value F max can be calculated using the following formula (2) using the average edge intensities F1, F2, and F3 and the distances z1, z2, and z3 to the respective focal planes.
Equation
[0039] The control device 30 specifies the three-dimensional position of the manipulator 16 based on the operation of the manipulator driving mechanism 26. The control device 30 first specifies the three-dimensional position of the manipulator 16 using a second coordinate system based on the driving axis of the manipulator 16. The three-dimensional position of the manipulator 16 based on the second coordinate system can be relatively specified by calculating the amount of movement of the manipulator 16 in the three-dimensional direction from the rotation angle of the motor of the manipulator driving mechanism 26, the driving amount of the actuator, etc. The second coordinate system is set for each of the holding pipette 16a and the injection pipette 16b, for example, and the three-dimensional positions of the tips 38a, 38b of the holding pipette 16a and the injection pipette 16b are specified in each second coordinate system. The second coordinate system may be a common coordinate system for the holding pipette 16a and the injection pipette 16b.
[0040] The control device 30 identifies the three-dimensional position of the manipulator 16 in the first coordinate system by performing coordinate conversion from the second coordinate system to the first coordinate system. As described above, the first coordinate system is a coordinate system based on the optical axis A of the objective lens 20. If the position coordinate in the first coordinate system is P1 (X1, Y1, Z1) and the position coordinate in the second coordinate system is P2 (X2, Y2, Z2), the coordinate conversion from the second coordinate system to the first coordinate system can be expressed by the following formula (3).
number
[0041] In the above formula (3), r ij is a transformation parameter that indicates the rotation from the second coordinate system to the first coordinate system, and t i is a transformation parameter that indicates the translation from the origin of the second coordinate system to the origin of the first coordinate system. The coordinate transformation parameter r ij and t ican be derived from the correlation between the position coordinates of the tip of the manipulator 16 measured in the second coordinate system and the position coordinates of the tip of the manipulator 16 measured in the first coordinate system. Specifically, the tip of the manipulator 16 is placed at a plurality of measurement positions, about 100 to 200 different from each other, and the position coordinates of the tip of the manipulator 16 at each measurement position are measured in the first coordinate system and the second coordinate system, respectively. After that, the correlation between the position coordinates in the first coordinate system and the second coordinate system is specified using the least squares method, whereby the coordinate transformation parameter r ij and t i can be estimated.
[0042] The position coordinates in the first coordinate system can be determined from the working distance WD of the objective lens 20 when capturing an image of the tip of the manipulator 16 and the coordinates (u, v) of the pixel of the tip of the manipulator 16 in the captured image. The relationship between the position coordinates P1 (X1, Y1, Z1) in the first coordinate system and the coordinates (u, v) of the pixel of the captured image can be expressed by the following formula (4) using perspective projection transformation.
number
number
[0043] FIG. 5 is a diagram showing a schematic diagram of a measurement pattern of the manipulator 16 for deriving coordinate transformation parameters. In a first coordinate system based on the optical axis A of the objective lens 20, a plurality of measurement planes 60a, 60b, and 60c are set at a distance δ in the z direction, and a plurality of measurement positions 62a, 62b, and 62c are set in a lattice pattern in each of the measurement planes 60a to 60c. In the illustrated example, three measurement planes 60a to 60c are set, and 9×6=54 measurement positions (for example, a plurality of first measurement positions 62a) are set in one measurement plane (for example, the first measurement plane 60a). Note that the number of measurement planes and the number of measurement positions are not limited to these. The number of measurement planes may be four or more, and the number of measurement positions in each measurement plane may be more or less than 54.
[0044] First, the manipulator 16 is moved so that the tip 38 of the manipulator 16 is positioned within the field of view of the objective lens 20. Next, the working distance WD of the objective lens 20 is adjusted so that the tip 38 of the manipulator 16 is focused. This determines the distance WD to the first measurement plane 60a. Next, with the tip 38 of the manipulator 16 in focus, the manipulator 16 is moved, and the position coordinates of the tip 38 of the manipulator 16 are identified in each of the first coordinate system and the second coordinate system at multiple measurement positions 62a within the first measurement plane 60a.
[0045] When the measurement at all the measurement positions 62a in the first measurement plane 60a is completed, the tip 38 of the manipulator 16 is then moved to the second measurement plane 60b. At this time, the tip 38 of the manipulator 16 is moved by δ in the z direction, and the working distance is increased by δ to WD+δ. The distance δ between the first measurement plane 60a and the second measurement plane 60b is set to a value larger than the depth of field of the objective lens 20, for example, about 5 to 10 times the depth of field. For example, the depth of field of the objective lens 20 is 30 μm, and δ=200 μm. Therefore, the multiple measurement positions 62a, 62b, and 62c in the measurement pattern in FIG. 5 are distributed over a range wider than the depth of field of the objective lens 20 in the optical axis direction (z direction) of the objective lens 20.
[0046] In the second measurement plane 60b, as in the first measurement plane 60a, the manipulator 16 is moved with the tip 38 of the manipulator 16 in focus. As a result, the position coordinates of the tip 38 of the manipulator 16 are identified in each of the first and second coordinate systems at multiple measurement positions 62b in the second measurement plane 60b. When measurements at all measurement positions 62b in the second measurement plane 60b are completed, the manipulator 16 is moved by δ in the z direction so that the tip 38 of the manipulator 16 is located in the third measurement plane 60c, and the working distance WD is increased by δ to WD+2δ. After that, the position coordinates of the tip 38 of the manipulator 16 are identified in each of the first and second coordinate systems at multiple measurement positions 62c in the third measurement plane 60c.
[0047] After identifying the position coordinates at all the measurement positions 62a, 62b, and 62c, the control device 30 calculates the coordinate transformation parameter r using the above equation (5). ij and t i The control device 30 performs measurements separately on the tips 38a, 38b of the holding pipette 16a and the injection pipette 16b to determine the coordinate transformation parameter r ij and t i The coordinate transformation parameter r ij and t i Once determined, the coordinate transformation parameters r can be used continuously unless the device configuration is changed. ij and t i The control device 30 does not need to determine the coordinate transformation parameters r ij and t i The stored coordinate transformation parameter r ij and t i may be used to transform the position coordinates of the tip 38 of the manipulator 16.
[0048] Coordinate transformation parameter r ij and t iconverts the relative movement of the manipulator 16 in the second coordinate system into the relative movement in the first coordinate system with high accuracy. On the other hand, the coordinate conversion parameter r ij and t i According to the inventor's knowledge, when the coordinate points of the second coordinate system are transformed into the coordinate points of the first coordinate system using the coordinate transformation parameter r ij and t i However, the accuracy of the absolute value of the position coordinates is low. ij and t i is determined by utilizing the property of perspective projection transformation that the closer an imaged object is to the viewpoint, the larger it appears, and the farther it is from the viewpoint, the smaller it appears. In the case of a normal camera, the depth of field of the camera is relatively large, so that the apparent size of the object can be significantly changed by shifting the position of the object in the depth direction within the depth of field of the camera. On the other hand, in the case of a microscope, the depth of field of the objective lens 20 is small, so that the apparent size of the object hardly changes even if the position of the object in the depth direction is shifted within the depth of field of the objective lens 20. In other words, the apparent change in the captured image when the z-direction coordinate of the tip 38 of the manipulator 16 is changed is small, and if the z-direction coordinate of the first coordinate system is estimated based on the captured image, the deviation from the actual position (true value) becomes large. Therefore, in the present disclosure, similar to the method of determining the position coordinate in the z direction of the sample 40, the absolute value of the z coordinate of the tip 38 of the manipulator 16 is determined based on the average edge strength F of the captured image of the tip 38 of the manipulator 16, and the absolute value of the position coordinate in the first coordinate system (initial value or origin) is calibrated.
[0049] A method for calibrating the origin of the manipulator 16 will be described below. First, the control device 30 positions the tip 38 of the manipulator 16 at an initial point (temporary origin) O that is set within the field of view of the objective lens 20. The control device 30 calculates the position coordinates (X 20 ,Y 20 ,Z 20 ) is transformed to the position coordinate (X 10 ,Y 10 ,Z10 ) is calculated. The control device 30 captures an image of the tip 38 of the manipulator 16 located at the initial point O while changing the working distance WD to generate a plurality of captured images. It is preferable that the variation width of the working distance WD is as small as possible, and is preferably equal to or less than the depth of field of the objective lens 20 (for example, about 1 μm). The control device 30 calculates the average edge strength F of the tip 38 of the manipulator 16 contained in each of the plurality of captured images, and identifies the captured image in which the average edge strength F is maximum. The control device 30 calculates the working distance WD when the captured image in which the average edge strength F is maximum is captured. max is the true value of the z coordinate of the initial point O. Here, if the z coordinate of the tip 38 of the manipulator 16 in the first coordinate system before calibration is Z1 and the z coordinate after calibration is Z1', then Z1'=Z1-Z 10 +WD max This is expressed as:
[0050] During use of the manipulation system 10, the control device 30 determines in real time the three-dimensional position of the sample 40 in the field of view of the objective lens 20 in the first coordinate system, and also determines in real time the three-dimensional position of the tip 38 of the manipulator 16 in the first coordinate system. The three-dimensional position of the sample 40 can be determined by changing the working distance WD and acquiring at least three captured images. The time required to determine the three-dimensional position of the sample 40 is mainly restricted by the time required to drive the variable focus lens 22 and change the working distance WD. The time required to change the working distance WD depends on the specifications of the variable focus lens 22 and the amount of change in the working distance WD, but is, for example, about 10 milliseconds to 20 milliseconds. Therefore, the time required to change the working distance WD and acquire three captured images is about 30 milliseconds to 60 milliseconds, and the three-dimensional position of the sample 40 can be captured at a cycle of 16 to 33 times per second. Meanwhile, the three-dimensional position of the tip 38 of the manipulator 16 can be sequentially captured based on the operation of the manipulator drive mechanism 26 and can be determined within the time required to determine the three-dimensional position of the sample 40 .
[0051] The control device 30 maps the three-dimensional arrangement of the sample 40 and the manipulator 16 onto a virtual space based on the identified three-dimensional positions of the sample 40 and the manipulator 16. The range of the mapped virtual space is wider than the range of the field of view of the objective lens 20. The three-dimensional position of the sample 40 can be identified only within the field of view of the objective lens 20, but the three-dimensional position of the manipulator 16 can be identified both within and outside the field of view of the objective lens 20. Therefore, in the virtual space, the position of the tip 38 of the manipulator 16 can be constantly mapped even if the tip 38 of the manipulator 16 is not located within the field of view of the objective lens 20. The three-dimensional arrangement of the sample 40 and the manipulator 16 mapped onto the virtual space is displayed on the three-dimensional display device 32 in real time. The display cycle of the three-dimensional display device 32 is, for example, 40 milliseconds (25 frames per second).
[0052] 6(a)-(j) are diagrams showing captured images within the field of view of the objective lens 20 and examples of displays on the three-dimensional display device 32, showing a time series of a flow in which the sample 40 is fixed by the holding pipette 16a from the left side and the sample 40 is manipulated by the injection pipette 16b from the right side. FIGS. 6(a)-(e) are captured images (i.e., microscope images) within the field of view of the objective lens 20, and FIGS. 6(f)-(j) are examples of displays on the three-dimensional display device 32 corresponding to FIGS. 6(a)-(e), respectively. A first object 70 imitating the sample 40, a second object 72 imitating the holding pipette 16a, and a third object 74 imitating the injection pipette 16b are three-dimensionally displayed on the three-dimensional display device 32.
[0053] 6(a) and (b) show a state before the injection pipette 16b is brought close to the sample 40, and the injection pipette 16b is not captured in the captured image. However, the three-dimensional position of the injection pipette 16b can be captured even if it is outside the field of view of the objective lens 20. Therefore, a third object 74 that imitates the injection pipette 16b is also displayed on the three-dimensional display device 32 in FIGS. 6(f) and (g). As a result, even if the injection pipette 16b is not visible within the field of view of the objective lens 20, the injection pipette 16b can be operated so that the tip of the injection pipette 16b approaches the sample 40 while looking at the three-dimensional display device 32.
[0054] The three-dimensional display device 32 can arbitrarily switch the viewpoint, for example, to a top view of the sample 40 when viewed in the z direction, a side view of the sample 40 when viewed in the x direction, or a side view of the sample 40 when viewed in the y direction. Fig. 6(f) is an example of a top view of the sample 40 when viewed in the z direction, and Fig. 6(g) is an example of a side view of the sample 40 when viewed in the y direction. By appropriately switching the viewpoint, the insertion angle of the manipulator 16 relative to the sample 40 can be easily confirmed.
[0055] 6(c) and (d) show the state in which the injection pipette 16b is in contact with the sample 40 and being operated. Looking at the captured image in FIG. 6(c), it appears that the injection pipette 16b is stuck into the sample 40. However, looking at the three-dimensional display device 32 in FIG. 6(h), it can be seen that the injection pipette 16b has slipped under the sample 40 and is not stuck into the sample 40. It can also be seen that the sample 40 is being lifted upward by the injection pipette 16b, and that the sample 40 is about to become loose from being fixed by the holding pipette 16a.
[0056] Fig. 6(e) shows a state in which the injection pipette 16b is inserted into the sample 40 and the sample 40 is separated from the holding pipette 16a. At this time, by looking at the three-dimensional display device 32 in Fig. 6(i), it is possible to confirm how deep into the sample 40 the injection pipette 16b has been inserted. In addition, by switching the viewpoint of the three-dimensional display device 32, it is possible to confirm whether the injection pipette 16b has been inserted near the center of the sample 40.
[0057] Next, the flow of operations of the manipulation system 10 will be described. Fig. 7 is a flowchart showing a three-dimensional position presentation method according to the embodiment. First, the origin of the manipulator 16 is calibrated (S10). The origin of the manipulator 16 is calibrated by changing the working distance and capturing an image of the tip 38 of the manipulator 16, and determining the working distance WD at which the average edge strength F of the captured image is maximized. max The process is executed by identifying the sample 40. Next, an image of the observation area is acquired (S12), and if the sample 40 is present in the image (Y in S14), a plurality of images are acquired by changing the working distance WD (S16). The three-dimensional position of the sample 40 is identified based on the maximum value when the average edge intensity F of the sample 40 included in the plurality of images is approximated by Gaussian distribution (S18). Next, the three-dimensional position of the manipulator 16 in the second coordinate system is identified based on the operation of the manipulator 16, and the three-dimensional position of the manipulator 16 in the first coordinate system is identified by coordinate conversion from the second coordinate system to the first coordinate system (S20). Based on the identified three-dimensional positions of the sample 40 and the manipulator 16 in the first coordinate system, the sample 40 and the manipulator 16 are mapped in a virtual space (S22), and the three-dimensional positional relationship of the sample 40 and the manipulator 16 mapped in the virtual space is displayed on the three-dimensional display device 32 or the like (S24). If there is no specimen 40 in S14 (N in S14), the processes of S16 to S24 are skipped. If the use of the manipulation system 10 is to be continued (N in S26), the processes of S12 to S24 are repeated. If the use of the manipulation system 10 is to be ended (Y in S26), this flow is ended.
[0058] According to an example of the present disclosure, the processes of S12 to S24 are repeated every 40 milliseconds, and the three-dimensional positions of the sample 40 and the manipulator 16 are updated and displayed at a cycle of 25 frames per second. Since this display cycle is almost the same as the frame rate of a general video, the three-dimensional positions of the sample 40 and the manipulator 16 appear to be reflected immediately without a time lag to a user viewing the three-dimensional display device 32. As a result, the user can accurately operate the sample 40 with the manipulator 16 while viewing the three-dimensional display device 32 without feeling stress caused by a discrepancy between the actual position and the displayed position. Since the three-dimensional display device 32 presents the positional relationship between the sample 40 and the manipulator 16 in a state where stereoscopic vision is possible, the positional relationship between the sample 40 and the manipulator 16 can be confirmed from various angles by changing the direction of the line of sight with respect to the three-dimensional display device 32. In other words, the enlarged sample 40 and the manipulator 16 can be displayed as if they exist three-dimensionally in front of the user's eyes, making it easy to grasp the three-dimensional positional relationship between the two. This makes it possible to improve the operability when precisely manipulating a minute sample 40 such as a cell, and to increase the convenience of the manipulation system 10.
[0059] The present disclosure has been described above based on the embodiments. It will be understood by those skilled in the art that the present disclosure is not limited to the above embodiments, that various design changes are possible, that various modifications are possible, and that such modifications are also within the scope of the present disclosure. [Explanation of symbols]
[0060] 10...manipulation system, 12...stage, 16...manipulator, 16a...holding pipette, 16b...injection pipette, 20...objective lens, 22...variable focus lens, 24...imaging device, 26...manipulator drive mechanism, 28...lens drive mechanism, 30...control device, 32...three-dimensional display device, 38...tip, 40...sample, 42...observation light, 46...sample dish, 48...liquid.
Claims
1. A stage for supporting the sample; a manipulator for manipulating the sample on the stage; a manipulator drive mechanism for moving the manipulator; an objective lens for observing the sample and the manipulator; a variable focus lens onto which the observation light that has passed through the objective lens is incident; an imaging device that captures the observation light that has passed through the variable focus lens to generate a captured image; a lens driving mechanism that changes the refractive power of the focus variable lens to change a working distance from a tip of the objective lens to a focal position of the observation light; a control device that identifies a three-dimensional position of the sample based on a plurality of captured images captured while changing the working distance, identifies a three-dimensional position of the manipulator based on an operation of the manipulator drive mechanism, and maps a three-dimensional positional relationship between the sample and the manipulator in a virtual space; a display device that displays a three-dimensional arrangement relationship between the sample and the manipulator mapped in the virtual space.
2. The manipulation system according to claim 1, characterized in that the control device identifies a three-dimensional position of the sample using a first coordinate system based on the optical axis of the objective lens, identifies a three-dimensional position of the manipulator in the first coordinate system by coordinate transformation from a second coordinate system based on a drive axis of the manipulator to the first coordinate system, and maps a three-dimensional positional relationship between the sample and the manipulator into the virtual space using the first coordinate system.
3. coordinate transformation parameters used for coordinate transformation from the second coordinate system to the first coordinate system are determined based on a plurality of coordinate values in the second coordinate system of the tip of the manipulator when the tip of the manipulator is moved to a plurality of measurement positions, and a plurality of coordinate values in the first coordinate system of the tip of the manipulator specified based on a captured image capturing the tip of the manipulator when the manipulator is moved to the plurality of measurement positions, 3. The manipulation system according to claim 2, wherein the plurality of measurement positions are distributed over a range wider than a depth of field of the objective lens in the optical axis direction of the objective lens.
4. The manipulation system according to any one of claims 1 to 3, characterized in that the control device calculates an average edge intensity of the sample contained in the plurality of captured images captured while changing the working distance, and identifies a three-dimensional position of the sample based on a maximum value obtained by Gaussian distribution approximation of a correlation between the working distance and the average edge intensity.
5. The manipulation system according to any one of claims 1 to 4, characterized in that the control device calculates an average edge strength of the tip of the manipulator contained in a plurality of captured images taken while changing the working distance, and calibrates the three-dimensional position of the manipulator based on the working distance when the average edge strength is maximum.
6. The manipulation system according to any one of claims 1 to 5, characterized in that the control device calculates the working distance based on a focal length of the objective lens, a focal length of the variable-focus lens, a distance between the objective lens and the variable-focus lens, and a refractive index distribution of an optical path from a tip of the objective lens to the sample.
7. changing the refractive power of a variable-focus lens onto which observation light that has passed through an objective lens for observing a sample and a manipulator on a stage is incident, thereby changing a working distance from a tip of the objective lens to a focal position of the observation light; capturing the observation light with an imaging device while changing the working distance to generate a plurality of captured images; Identifying a three-dimensional position of the sample based on the plurality of captured images; determining a three-dimensional position of the manipulator based on an operation of a manipulator drive mechanism that moves the manipulator; mapping a three-dimensional arrangement relationship between the sample and the manipulator into a virtual space; and displaying a three-dimensional position relationship between the sample and the manipulator mapped in the virtual space.
8. a function of controlling a lens drive mechanism for changing the refractive power of a variable-focus lens onto which observation light passes through an objective lens for observing a sample and a manipulator on a stage, and changing a working distance from a tip of the objective lens to a focal position of the observation light; a function of acquiring a plurality of captured images by capturing the observation light using an imaging device while changing the working distance; a function of identifying a three-dimensional position of the sample based on the plurality of captured images; A function of identifying a three-dimensional position of the manipulator based on an operation of a manipulator drive mechanism that moves the manipulator; and a function of mapping a three-dimensional arrangement relationship of the sample and the manipulator onto a virtual space.
Citation Information
Patent Citations
Laser micro-dissection method for extraction of microscopic object from biological preparation using image analysis technique for automatic object identifcation and marking of required separation line
DE102004051508A1
Micromanipulator device
JP1998180660A
Micro manipulation device for minute operation
JP2004255493A
Micromanipulation system
JP2005258413A
Focal plane inclination type cofocal surface profile measuring device
JP2008145268A
