Manipulation system and manipulation method

Abstract

To improve operability when manipulating samples using a manipulator. [Solution] The manipulation system 110 includes a manipulator 16 for manipulating a sample 40, a manipulator drive mechanism 26 for moving the manipulator 16, a pump 78 for variably controlling the suction and discharge forces at the tip of the manipulator 16, an input operation device 80 including a holding member 84 for the user to hold, a rotating member 86 that rotates relative to the holding member 84 by the user's gripping motion, and an angle sensor for detecting the rotation angle of the rotating member 86, and a control device 30 that controls the operation of the pump 78 based on angle information indicating the rotation angle of the rotating member 86 detected by the angle sensor.

Description

[Technical Field]

[0001] This disclosure relates to a manipulation system and a manipulation method. [Background technology]

[0002] Manipulation systems are known for manipulating cells using manipulators. Because cells are microscopic, manipulation of the cells is performed while observing the position of the cells and manipulators under a microscope. The manipulator is mounted on a table that allows for fine movement in three axes (X, Y, and Z), and the user moves the manipulator using a joystick or similar device. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-122898 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] To properly manipulate cells, it is necessary not only to control the position of the manipulator but also to properly control the force applied to the cells by the manipulator. When using a joystick to operate the manipulator, the user cannot directly perceive the force applied to the cells. Experienced cell manipulators can skillfully manipulate the cells by imagining the force applied based on visual information from the microscope. However, for beginners in cell manipulation, it is not easy to grasp the force applied to cells based solely on visual information, and it generally takes a long time (e.g., more than a year) to acquire the necessary skills.

[0005] This disclosure has been made in view of these challenges, and one of its exemplary objectives is to provide a technique for improving operability when manipulating a sample using a manipulator. [Means for solving the problem]

[0006] A manipulation system in one aspect of the present disclosure includes a manipulator for manipulating a sample, a manipulator drive mechanism for moving the manipulator, an imaging device for imaging the sample through an objective lens, a control device for generating force information indicating the magnitude of force to be presented to the user based on the image captured by the imaging device, and a force presentation device configured to receive input from the user to specify the position of the manipulator and to present force to the user in accordance with the force information generated by the control device.

[0007] Another aspect of the present disclosure is a manipulation method. This method comprises the steps of: acquiring position information from a force feedback device based on user input to specify the position of a manipulator; controlling the operation of a manipulator drive mechanism to move the manipulator based on the acquired position information; imaging a sample to be manipulated using the manipulator through an objective lens; generating force information indicating the magnitude of force to be presented to the user based on the captured image; and controlling the operation of the force feedback device so that force is presented to the user in accordance with the generated force information.

[0008] Yet another aspect of the present disclosure is a manipulation system. This manipulation system includes a manipulator for manipulating a sample, a manipulator drive mechanism for moving the manipulator, a pump for variably controlling the suction force and discharge force at the tip of the manipulator, a holding member held by a user, a link mechanism for supporting the holding member such that the position of the holding member is variable according to the operation of the user, a position sensor for detecting the position of the holding member, a rotating member that rotates with respect to the holding member by the gripping operation of the user, and an angle sensor for detecting the rotation angle of the rotating member. It also includes an input operation device and a control device that controls the operation of the manipulator drive mechanism based on the position information indicating the position of the holding member detected by the position sensor and controls the operation of the pump based on the angle information indicating the rotation angle of the rotating member detected by the angle sensor.

[0009] Yet another aspect of the present disclosure is a manipulation method. This method includes steps of obtaining position information indicating the position of a holding member held by a user and supported by a link mechanism such that the position is variable according to the operation of the user, obtaining angle information indicating the rotation angle of a rotating member that rotates with respect to the holding member by the gripping operation of the user, controlling the operation of a manipulator drive mechanism for moving a manipulator for manipulating a sample based on the obtained position information, and controlling the operation of a pump for variably controlling the suction force and discharge force at the tip of the manipulator based on the obtained angle information.

[0010] Any combination of the above components, or those obtained by mutually replacing the components and expressions of the present disclosure among methods, systems, programs, etc., are also effective as aspects of the present disclosure.

Advantages of the Invention

[0011] According to the present disclosure, the operability when operating a sample using a manipulator can be improved.

Brief Description of the Drawings

[0012] [Figure 1] This diagram schematically shows the configuration of the manipulation system according to the first embodiment. [Figure 2] This diagram schematically shows the working distance considering the effect of refractive index. [Figure 3] This diagram schematically illustrates a method for determining the z-coordinate of a sample. [Figure 4] This graph shows the correlation between average edge strength (approximated by a Gaussian distribution) and working distance. [Figure 5] This diagram schematically shows the manipulator's measurement pattern when deriving coordinate transformation parameters. [Figure 6] Figures 6(a) to 6(j) show examples of images captured within the field of view of the objective lens and their display on a three-dimensional display device. [Figure 7] This is a flowchart showing a three-dimensional position presentation method according to an embodiment. [Figure 8] Figures 8(a) to 8(c) schematically illustrate how a sample is manipulated with a manipulator. [Figure 9] This is a flowchart showing the force feedback method according to the embodiment. [Figure 10] This figure schematically shows the configuration of the manipulation system according to the second embodiment. [Figure 11] Figures 11(a) and 11(b) schematically show the configuration of the second force feedback device. [Figure 12] This diagram schematically illustrates the method for determining contact between the sample and the manipulator. [Figure 13] This diagram schematically illustrates how a sample is manipulated with a manipulator. [Figure 14] This diagram schematically illustrates how a sample is manipulated with a manipulator. [Figure 15] This diagram schematically illustrates how a sample is manipulated with a manipulator. [Figure 16] This flowchart shows an example of a manipulation method related to this disclosure. [Figure 17]This flowchart shows an example of a manipulation method related to this disclosure. [Figure 18] This figure schematically shows the configuration of the manipulation system according to the third embodiment. [Figure 19] This diagram schematically shows the configuration of the third force feedback device. [Figure 20] This diagram schematically shows the deformation amounts at multiple locations of a sample manipulated by a manipulator. [Figure 21] This diagram schematically illustrates a method for calculating the deformation of a sample using optical flow. [Figure 22] Figures 22(a) to (c) schematically illustrate the perforation operation of a sample using a manipulator. [Modes for carrying out the invention]

[0013] (First Embodiment) First, I will give an overview of this disclosure. This disclosure relates to a manipulation system. The manipulation system comprises an optical microscope for observing a sample such as cells and a manipulator for manipulating the sample. The user manipulates the sample by moving the manipulator while observing the sample and the manipulator with the optical microscope. The position of the manipulator is controlled by a drive mechanism such as an actuator to enable micro-level manipulation. In order to properly manipulate cells, it is necessary to properly control the force applied to the cells. However, the user cannot directly feel the force applied to the cells by the manipulator. Furthermore, the force applied to the cells by the manipulator is very small, about 1 μN to 1 mN (0.1 mgf to 0.1 gf), and it is difficult to detect the force in real time using sensors or the like.

[0014] This disclosure identifies in real time the amount of change in at least one of the position and shape of a sample manipulated by a manipulator, and estimates the force applied to the sample based on the identified amount of change. Furthermore, the estimated force is amplified by approximately 100 to 10,000 times and fed back through a force feedback device, allowing the user to feel the reaction force applied to the manipulator while manipulating the sample. According to this disclosure, it is possible to provide force feedback as if directly manipulating cells with one's hands, thereby improving the operability of the manipulation system. In particular, it can help beginners with little experience in cell manipulation to acquire skills.

[0015] Figure 1 is a schematic diagram showing the configuration of a manipulation system 10 according to the first embodiment. The manipulation system 10 has an inverted microscope configuration. The manipulation system 10 comprises a stage 12, an illumination device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator drive mechanism 26, a lens drive mechanism 28, a control device 30, a display device 32, an input device 34, and a force feedback device 36.

[0016] In Figure 1, a first coordinate system is set up with the optical axis A of the objective lens 20 on the stage 12 as the reference. The direction in which the optical axis A of the objective lens 20 extends on the stage 12 is defined as the z direction, and the directions perpendicular to the optical axis A are defined as the x and y directions. In the illustrated example, the direction in which the optical axis A of the objective lens 20 extends on the stage 12 coincides with the direction perpendicular to the support surface 12a of the stage 12. Note that the direction in which the optical axis A of the objective lens 20 extends on the stage 12 may be deviated from the direction perpendicular to the support surface 12a of the stage 12.

[0017] The stage 12 has a support surface 12a for horizontally supporting the sample 40 and an opening 12b for allowing observation light 42 from the sample 40 to pass through. The sample 40 to be manipulated is not particularly limited, but it can be human or animal cells. The sample 40 is contained in a sample dish 46 made of a transparent material such as resin or glass, and the sample dish 46 is placed on the stage 12. The sample 40 is suspended in a liquid 48 such as water contained in the sample dish 46.

[0018] The illumination device 14 is positioned above the stage 12 and illuminates the sample 40 on the stage 12. The illumination device 14 projects illumination light 44, such as white light, toward the sample 40. The illumination device 14 is configured to provide transmitted illumination. The illumination device 14 may also be capable of projecting illumination light 44 of a specific wavelength selected for fluorescence observation or the like. For example, the illumination device 14 projects illumination light 44 that results in a uniform illuminance distribution on the stage 12.

[0019] The manipulator 16 is mounted on the stage 12 and used to manipulate the sample 40. In the illustrated example, the manipulator 16 includes a holding pipette 16a and an injection pipette 16b. For example, cells are fixed using the holding pipette 16a, and cell manipulation such as gene introduction into the cells is performed using the injection pipette 16b. In the illustrated example, two manipulators are provided, but the number of manipulators may be one or three or more.

[0020] The folding mirror 18 is positioned directly below the aperture 12b of the stage 12. The folding mirror 18 is positioned to reflect the observation light 42 from the sample 40 toward the objective lens 20. In the illustrated example, the optical axis A of the objective lens 20 is folded back by the folding mirror 18, but the objective lens 20 may be positioned on the optical axis extending in the z direction without the folding mirror 18.

[0021] The objective lens 20 is positioned where the observation light 42 from the folding mirror 18 enters. The objective lens 20 is positioned away from the folding mirror 18 in the +x direction. It is desirable that the objective lens 20 has a relatively long working distance (WD). The specifications of the objective lens 20, such as magnification and working distance, are not particularly limited, but for example, an ultra-long working type objective lens with a working distance of 20 mm to 40 mm at magnifications of 10 to 50 times can be used.

[0022] The variable focus lens 22 is positioned where the observation light 42 that has passed through the objective lens 20 enters. The variable focus lens 22 is positioned between the objective lens 20 and the imaging device 24, for example, adjacent to or close to the objective lens 20. The variable focus lens 22 is configured so that its refractive power is variable within a predetermined range. The variable focus lens 22 may be a convex lens having only positive refractive power, a concave lens having only negative refractive power, or it may be configured to switch between positive and negative refractive power.

[0023] The variable-focus lens 22 is composed of, for example, a liquid lens, and its focal length is made variable by deforming a flexible transparent film that seals the liquid lens. The shape of the transparent film is controlled by changing the pressure applied to the transparent film. For example, the focal length of the variable-focus lens 22 can be electrically controlled using an electromagnetic actuator or a piezoelectric element. The variable-focus lens 22 is configured to vary the effective working distance by a combination of the objective lens 20 and the variable-focus lens 22 within a range of about 2 mm.

[0024] The imaging device 24 captures the observation light 42 that has passed through the variable-focus lens 22 and generates an image. The imaging device 24 has an imaging lens 24a and an image sensor 24b. The imaging lens 24a forms an image of the observation light 42 on the image sensor 24b. The image sensor 24b is an image sensor such as a CMOS sensor and is capable of generating an image at a high frame rate. The frame rate of the imaging device 24 is not particularly limited, but is preferably 100 frames per second or more, and more preferably 500 frames per second or more.

[0025] The objective lens 20, the variable focus lens 22, and the imaging device 24 are arranged along the optical axis A extending in the x-direction and are fixed, for example, to a lens barrel extending in the x-direction. An additional folding mirror (not shown) may be provided between the variable focus lens 22 and the imaging device 24, resulting in a configuration where the optical axis A is further folded.

[0026] The manipulator drive mechanism 26 moves the manipulator 16, making its three-dimensional position variable. In the illustrated example, the manipulator drive mechanism 26 includes a first drive mechanism 26a and a second drive mechanism 26b. The first drive mechanism 26a is configured to move the holding pipette 16a, making its three-dimensional position variable. The second drive mechanism 26b is configured to move the injection pipette 16b, making its three-dimensional position variable. The three-dimensional positions of the holding pipette 16a and the manipulator 16 can be controlled independently of each other.

[0027] The lens drive mechanism 28 drives the variable-focus lens 22 and changes the refractive power of the variable-focus lens 22. By changing the refractive power of the variable-focus lens 22, the lens drive mechanism 28 changes the effective working distance of the combination of the objective lens 20 and the variable-focus lens 22. Here, the effective working distance is the distance from the tip of the objective lens 20 to the focal position of the observation light 42, and the distance from the tip of the objective lens 20 to the focal plane where the image captured by the imaging device 24 is in focus.

[0028] The control device 30 controls the overall operation of the manipulation system 10. Hardware-wise, the control device 30 can be implemented using components and mechanical devices such as a computer's CPU and memory, while software-wise, it can be implemented using computer programs. For example, the control device 30 can be configured using a general-purpose personal computer.

[0029] The display device 32 includes a three-dimensional display device 32a and a two-dimensional display device 32b. The three-dimensional display device 32a displays the three-dimensional positions of the sample 40 and the manipulator 16 in 3D. The three-dimensional display device 32a is, for example, a hologram display such as Looking Glass, and is a display device that enables stereoscopic viewing without the use of 3D glasses or the like. The three-dimensional display device 32a displays computer graphic (CG) images that mimic the sample 40 and the manipulator 16. The three-dimensional display device 32a may also be a head-mounted virtual reality (VR) display device.

[0030] The two-dimensional display device 32b is a liquid crystal display or the like, and displays the captured images captured by the imaging device 24 in real time. The two-dimensional display device 32b may display the three-dimensional positions of the sample 40 and the manipulator 16, or it may display a rendered image generated by projecting the sample 40 and the manipulator 16, which are mapped into a virtual space, onto an arbitrary observation surface using perspective projection.

[0031] The input device 34 is a device for inputting operations to the control device 30 and operating the manipulator 16. A mouse or keyboard can be used as the means for inputting operations to the control device 30. A joystick can be used as the means for operating the manipulator 16. By using the input device 34 such as a joystick, the tip position of the manipulator 16 can be moved on the order of micrometers, allowing for precise manipulation of the sample 40.

[0032] The force feedback device 36 is a means for operating the manipulator 16 and is configured to accept input operations from the user to specify the position of the manipulator 16. The force feedback device 36 has a multi-joint arm 36a and is configured so that the tip 36b of the multi-joint arm 36a is movable in three axes: X, Y, and Z. The user performs an input operation to specify the three-dimensional position of the manipulator 16 by grasping and moving the tip 36b of the multi-joint arm 36a. The force feedback device 36 has, for example, a sensor for determining the position of the tip 36b of the multi-joint arm 36a and transmits position information based on the position of the tip 36b of the multi-joint arm 36a to the control device 30.

[0033] The force feedback device 36 also serves as a force feedback means for presenting to the user the force applied to the manipulator 16 during cell manipulation. The force feedback device 36 is configured to be able to present force feedback in the XYZ three axes at the tip 36b of the articulated arm 36a. The force feedback device 36 has an actuator for applying a reaction force to the articulated arm 36a and controls the operation of the actuator based on force information transmitted from the control device 30.

[0034] In this embodiment, an input device 34 such as a joystick is used to operate the holding pipette 16a, and a force feedback device 36 is used to operate the injection pipette 16b. Alternatively, the force feedback device 36 may be used to operate the holding pipette 16a.

[0035] This disclosure describes how to determine the three-dimensional positions of the sample 40 and the manipulator 16 in real time, and how to estimate the force acting between the sample 40 and the manipulator 16 using the determined three-dimensional positions. First, a method for determining the three-dimensional positions of the sample 40 and the manipulator 16 in real time will be described.

[0036] (Three-dimensional positioning method) The control device 30 identifies the three-dimensional position of the sample 40 included in the image based on the image captured by the imaging device 24. The control device 30 identifies the three-dimensional position of the sample 40 using a first coordinate system with the optical axis A of the objective lens 20 as the reference. The control device 30 identifies the sample 40 included in the image using image recognition technology and identifies the x and y coordinates of the sample 40 from the center position of the sample 40 in the image. The control device 30 identifies the z-coordinate of the sample 40 based on the working distance when imaging the sample 40. For example, the control device 30 sets the tip of the objective lens 20 as the z-origin (z=0) and the working distance from the objective lens 20 to the sample 40 as the z-coordinate of the sample 40.

[0037] The working distance WD can be calculated based on the focal length f1 of the objective lens 20, the focal length f2 of the variable-focus lens 22, the distance d between the objective lens 20 and the variable-focus lens 22, and the refractive index distribution of the optical path from the objective lens 20 to the sample 40. The combined focal length f0 of the combination of the objective lens 20 and the variable-focus lens 22 is expressed as f0 = f1(f2-d) / (f1+f2-d). The combined focal length f0 corresponds to the working distance when the optical path from the objective lens 20 to the sample 40 is air and the refractive index is approximately 1. In reality, the optical path from the objective lens 20 to the sample 40 contains a sample dish 46 or liquid 48. Therefore, the actual working distance WD deviates from the combined focal length f0 due to the influence of the refractive index of the sample dish 46 or liquid 48.

[0038] Figure 2 schematically shows the working distance WD considering the effect of refractive index. As shown in the figure, the presence of the sample dish 46 and liquid 48 in the optical path from the objective lens 20 to the sample 40 causes the observation light 42 incident on the objective lens 20 to be refracted. As a result, the actual working distance WD is longer than the combined focal length f0 of the objective lens 20 and the variable focus lens 22 when the sample dish 46 and liquid 48 are not present. The actual working distance WD can be expressed by the following equation (1), using the combined focal length f0, the refractive index n0 of air, the refractive index n1 of the sample dish 46, the refractive index n2 of the liquid 48, the distance a from the objective lens 20 to the sample dish 46, the thickness b of the sample dish 46, and the effective radius r assuming the objective lens 20 is an ideal plano-convex lens. Equation (1) can be derived based on the geometric relationships based on Snell's law.

number

[0039] As an example of an actual working distance WD, if the combined focal length f0 = 21.059 mm, the refractive index n1 = 1.592 of the polystyrene (PS) sample dish 46, the refractive index n2 = 1.33 of the pure water liquid 48, the distance a = 19.135 mm, the thickness b = 1.0 mm, and the effective radius r = 5.0 mm, then the working distance WD = 21.894 mm. In this case, the difference between the working distance WD and the combined focal length f0 is 0.835 mm, which is very large compared to the depth of field of the objective lens 20 (approximately 0.03 mm) and the size of the cells (approximately 0.1 mm). Therefore, by correcting the working distance WD considering the effect of the refractive index distribution from the objective lens 20 to the sample 40, the z-direction position coordinate of the sample 40 can be accurately determined.

[0040] The control device 30 controls the lens drive mechanism 28 to change the working distance WD. The control device 30 identifies the z-direction position coordinates of the sample 40 based on a plurality of captured images obtained by changing the working distance WD. Specifically, the average edge intensity F of the sample 40 included in each of the plurality of captured images is calculated, and the z-direction position coordinates of the sample 40 are identified based on the maximum value when the correlation between the working distance WD and the average edge intensity F is approximated by a Gaussian distribution. The average edge intensity F is obtained by calculating the edge intensity f(x, y) of each pixel of the captured image and averaging the edge intensities f(x, y) of all the pixels in the region where the sample 40 is included. When the luminance value of each pixel is I(x, y), the edge intensity is f(x, y) = {(I(x + 1, y) - I(x, y)) 2 + (I(x, y + 1) - I(x, y)) 2} 1 / 2 and is represented as such. The average edge intensity F indicates the contrast of the sample 40 included in the captured image, and the larger the average edge intensity F, the more the sample 40 is imaged with high contrast in a focused state.

[0041] FIG. 3 is a diagram schematically showing a method for identifying the z-direction position coordinates of the sample 40. In FIG. 3, the working distance WD of the objective lens 20 is set to the first distance z1, the second distance z2, and the third distance z3 (z1 < z2 < z3), and the captured images 52a, 52b, and 52c of the focal planes 50a, 50b, and 50c located at each of the distances z1 to z3 are schematically shown. In the illustrated example, the central coordinate z of the sample 40 is located between the first distance z1 and the second distance z2, showing a state where the sample 40 intersects the second focal plane 50b. Since the second focal plane 50b intersects the sample 40, the second captured image 52b of the second focal plane 50b includes the sample 40 in a focused state. On the other hand, the first captured image 52a of the first focal plane 50a away from the sample 40 includes the sample 40 in a slightly blurred state. Further, the third captured image 52c of the third focal plane 50c farther away from the sample 40 includes the sample 40 in an out-of-focus state. The interval Δz between the first distance z1, the second distance z2, and the third distance z3 is set to a value about 1 to 2 times the size of the sample 40. For example, if the size of the sample 40 is about 100 μm, then Δz = about 100 μm to 200 μm.

[0042] The control device 30 acquires a plurality of captured images 52a to 52c captured by changing the operating distance WD, and calculates the average edge intensities F1, F2, and F3 of the regions 54a to 54c in which the sample 40 is included in each of the captured images 52a to 52c. In the example shown in FIG. 3, the magnitude relationship of the average edge intensities of each of the captured images 52a to 52c is F3 < F1 < F2. The control device 30 approximates the correlation between the calculated average edge intensities F1, F2, and F3 and the distances z1, z2, and z3 to the respective focal planes 50a to 50c with a Gaussian distribution.

[0043] FIG. 4 is a graph showing the relationship between the average edge intensities F1, F2, and F3 approximated by a Gaussian distribution and the operating distance WD. As shown in the figure, if the magnitude relationship of the average edge intensities is F1 < F2 and F3 < F2, the average edge intensities F1 to F3 are approximated by a Gaussian distribution and the maximum value F max can be specified. The maximum value F max corresponds to the position where the focus of the sample 40 is the best, so the operating distance z0 corresponding to the maximum value F max can be regarded as the z coordinate of the center of the sample 40. Note that the operating distance z0 corresponding to the maximum value F max can be calculated using the following formula (2) using the average edge intensities F1, F2, F3 and the distances z1, z2, z3 to the respective focal planes.

Equation

[0044] The control device 30 identifies the three-dimensional position of the manipulator 16 based on the operation of the manipulator drive mechanism 26. First, the control device 30 identifies the three-dimensional position of the manipulator 16 using a second coordinate system based on the drive shafts of the manipulator 16. The three-dimensional position of the manipulator 16 based on the second coordinate system can be relatively identified by calculating the amount of movement of the manipulator 16 in the three-dimensional directions from the rotation angle of the motor of the manipulator drive mechanism 26, the drive amount of the actuator, and the like. The second coordinate system is set for each of the holding pipette 16a and the injection pipette 16b, for example, and the three-dimensional positions of the tips 38a and 38b of the holding pipette 16a and the injection pipette 16b are identified in their respective second coordinate systems. The second coordinate system may be a common coordinate system for the holding pipette 16a and the injection pipette 16b.

[0045] The control device 30 identifies the three-dimensional position of the manipulator 16 in the first coordinate system by performing a coordinate transformation from the second coordinate system to the first coordinate system. As described above, the first coordinate system is a coordinate system based on the optical axis A of the objective lens 20. If the position coordinates in the first coordinate system are P1(X1, Y1, Z1) and the position coordinates in the second coordinate system are P2(X2, Y2, Z2), the coordinate transformation from the second coordinate system to the first coordinate system can be expressed by the following equation (3).

Equation

[0046] In the above equation (3), r ij is a transformation parameter indicating the rotation from the second coordinate system to the first coordinate system, and t i is a transformation parameter indicating the translation from the origin of the second coordinate system to the origin of the first coordinate system. The coordinate transformation parameters r ij and t iThis can be derived from the correlation between the position coordinates of the tip of the manipulator 16 measured in the second coordinate system and the position coordinates of the tip of the manipulator 16 measured in the first coordinate system. Specifically, the tip of the manipulator 16 is placed at approximately 100 to 200 different measurement positions, and the position coordinates of the tip of the manipulator 16 at each measurement position are measured in both the first and second coordinate systems. Subsequently, the correlation between the position coordinates in the first and second coordinate systems is identified using the least squares method, thereby determining the coordinate transformation parameter r ij and t i It is possible to estimate this.

[0047] The position coordinates in the first coordinate system can be determined from the working distance WD of the objective lens 20 when imaging the tip of the manipulator 16 and the coordinates (u,v) of the pixel at the tip of the manipulator 16 in the captured image. The relationship between the position coordinates P1(X1,Y1,Z1) in the first coordinate system and the coordinates (u,v) of the pixel in the captured image can be expressed by the following equation (4) using perspective projection transformation.

number

number

[0048] Figure 5 schematically shows the measurement pattern of the manipulator 16 for deriving coordinate transformation parameters. In a first coordinate system with respect to the optical axis A of the objective lens 20, multiple measurement surfaces 60a, 60b, and 60c are set at a distance δ in the z direction, and multiple measurement positions 62a, 62b, and 62c are set in a grid within each measurement surface 60a to 60c. In the illustrated example, three measurement surfaces 60a to 60c are set, and 9 × 6 = 54 measurement positions (for example, multiple first measurement positions 62a) are set within one measurement surface (for example, the first measurement surface 60a). Note that the number of measurement surfaces and the number of measurement positions are not limited to these. The number of measurement surfaces may be 4 or more, and the number of measurement positions within each measurement surface may be more or less than 54.

[0049] First, the manipulator 16 is moved so that its tip 38 is within the field of view of the objective lens 20. Next, the working distance WD of the objective lens 20 is adjusted so that the tip 38 of the manipulator 16 is in focus. This determines the distance WD to the first measurement surface 60a. Subsequently, with the tip 38 of the manipulator 16 in focus, the manipulator 16 is moved to determine the position coordinates of the tip 38 of the manipulator 16 in the first and second coordinate systems at multiple measurement positions 62a within the first measurement surface 60a.

[0050] Once measurements are completed at all measurement positions 62a within the first measurement surface 60a, the tip 38 of the manipulator 16 is moved to the second measurement surface 60b. At this time, the tip 38 of the manipulator 16 is moved by δ in the z direction, and the working distance is increased by δ to WD+δ. The distance δ between the first measurement surface 60a and the second measurement surface 60b is set to a value greater than the depth of field of the objective lens 20, for example, about 5 to 10 times the depth of field. For example, if the depth of field of the objective lens 20 is 30 μm, then δ = 200 μm. Therefore, the multiple measurement positions 62a, 62b, and 62c in the measurement pattern of Figure 5 are distributed over a wider range than the depth of field of the objective lens 20 in the optical axis direction (z direction) of the objective lens 20.

[0051] On the second measurement surface 60b, as with the first measurement surface 60a, the manipulator 16 is moved while the focus is on the tip 38 of the manipulator 16. This determines the position coordinates of the tip 38 of the manipulator 16 in both the first and second coordinate systems at multiple measurement positions 62b within the second measurement surface 60b. Once measurements are completed at all measurement positions 62b within the second measurement surface 60b, the manipulator 16 is moved by δ in the z direction so that the tip 38 of the manipulator 16 is positioned on the third measurement surface 60c, and the working distance WD is increased by δ to WD+2δ. Subsequently, the position coordinates of the tip 38 of the manipulator 16 are determined in both the first and second coordinate systems at multiple measurement positions 62c within the third measurement surface 60c.

[0052] After the control device 30 identifies the position coordinates at all measurement positions 62a, 62b, and 62c, it uses the above equation (5) to determine the coordinate transformation parameter r ij and t i The control device 30 separately measures the tips 38a and 38b of the holding pipette 16a and injection pipette 16b and determines the coordinate transformation parameter r. ij and t i The coordinate transformation parameter r may be determined. ij and t i Once determined, it can be used continuously as long as the device configuration does not change. In other words, the control device 30 determines the coordinate transformation parameter r each time the manipulation system 10 is used. ij and t i It is not necessary to determine the coordinate transformation parameter r. ij and t i The stored coordinate transformation parameter r ij and t i The position coordinates of the tip 38 of the manipulator 16 may be transformed using this method.

[0053] Coordinate transformation parameter r ij and t iThis accurately converts the relative movement of the manipulator 16 in the second coordinate system to its relative movement in the first coordinate system. Meanwhile, the coordinate transformation parameter r ij and t i According to the inventor's knowledge, when a coordinate point in the second coordinate system is transformed to a coordinate point in the first coordinate system using , the coordinate points in both systems are found to be shifted. In other words, the coordinate transformation parameter r ij and t i Using only this method results in low precision in the absolute value of the position coordinates. The coordinate transformation parameter r mentioned above... ij and t i This is determined by utilizing the property of perspective projection transformation, which states that objects appear larger the closer they are to the viewpoint and smaller the farther they are. In the case of a normal camera, because the depth of field of the camera is relatively large, the apparent size of an object can be significantly altered by shifting its position in the depth direction within the depth of field of the camera. On the other hand, in the case of a microscope, because the depth of field of the objective lens 20 is small, even if the position of an object in the depth direction within the depth of field of the objective lens 20 is shifted, the apparent size of the object hardly changes. In other words, the apparent change in the captured image when the z-coordinate of the tip 38 of the manipulator 16 is changed is small, and when the z-coordinate of the first coordinate system is estimated based on the captured image, the deviation from the actual position (true value) becomes large. Therefore, in this disclosure, similar to the method for determining the z-direction position coordinates of the sample 40, the absolute value of the z-coordinate of the tip 38 of the manipulator 16 is determined based on the average edge intensity F of the captured image of the tip 38 of the manipulator 16, and the absolute value (initial value or origin) of the position coordinates in the first coordinate system is calibrated.

[0054] The following describes the calibration method for the origin of the manipulator 16. First, the control device 30 positions the tip 38 of the manipulator 16 at the initial point (provisional origin) O set within the field of view of the objective lens 20. The control device 30 sets the position coordinate (X) of the initial point O in the second coordinate system. 20 ,Y 20 ,Z 20 By transforming the coordinates of the initial point O, the position coordinates (X) in the first coordinate system are obtained. 10 ,Y 10 ,Z10 The control device 30 calculates the working distance WD of the manipulator 16 tip 38 located at the initial point O, and generates multiple images by changing the working distance WD. The range of change in the working distance WD is preferably as small as possible, and is preferably less than or equal to the depth of field of the objective lens 20 (for example, about 1 μm). The control device 30 calculates the average edge intensity F of the manipulator 16 tip 38 included in each of the multiple images, and identifies the image in which the average edge intensity F is maximized. The control device 30 calculates the working distance WD when the image in which the average edge intensity F is maximized is captured. max Let be the true value of the z-coordinate of the initial point O. Here, if we let Z1 be the z-coordinate of the tip 38 of the manipulator 16 in the first coordinate system before calibration, and Z1' be the z-coordinate after calibration, then Z1' = Z1 - Z 10 +WD max It is expressed as follows.

[0055] The control device 30, while using the manipulation system 10, determines the three-dimensional position of the sample 40 in the first coordinate system within the field of view of the objective lens 20 in real time, and also determines the three-dimensional position of the tip 38 of the manipulator 16 in the first coordinate system in real time. The three-dimensional position of the sample 40 can be determined by acquiring at least three images by changing the working distance WD. The time required to determine the three-dimensional position of the sample 40 is mainly limited by the time required to drive the variable focus lens 22 to change the working distance WD. The time required to change the working distance WD depends on the specifications of the variable focus lens 22 and the amount of change in the working distance WD, but for example, it is about 10 to 20 milliseconds. Therefore, the time required to acquire three images by changing the working distance WD is about 30 to 60 milliseconds, and the three-dimensional position of the sample 40 can be captured at a cycle of 16 to 33 times per second. On the other hand, the three-dimensional position of the tip 38 of the manipulator 16 can be sequentially captured based on the operation of the manipulator drive mechanism 26, and can be determined within the time required to determine the three-dimensional position of the sample 40.

[0056] Next, we will explain how to present the three-dimensional positions of the sample 40 and the manipulator 16.

[0057] (Three-dimensional position presentation method) The control device 30 maps the three-dimensional arrangement of the sample 40 and the manipulator 16 onto a virtual space based on the identified three-dimensional positions of the sample 40 and the manipulator 16. The range of the virtual space to be mapped is wider than the field of view of the objective lens 20. The three-dimensional position of the sample 40 can only be determined within the field of view of the objective lens 20, but the three-dimensional position of the manipulator 16 can be determined both within and outside the field of view of the objective lens 20. Therefore, in the virtual space, the position of the tip 38 of the manipulator 16 can be mapped at all times, even if the tip 38 of the manipulator 16 is not located within the field of view of the objective lens 20. The three-dimensional arrangement of the sample 40 and the manipulator 16 mapped onto the virtual space is displayed in real time on the three-dimensional display device 32a. The display cycle of the three-dimensional display device 32a is, for example, 40 milliseconds (25 frames per second).

[0058] Figures 6(a) to 6(j) show images captured within the field of view of the objective lens 20 and examples of the display on the three-dimensional display device 32a. They show the process of fixing the sample 40 with the holding pipette 16a from the left and manipulating the sample 40 with the injection pipette 16b from the right, in chronological order. Figures 6(a) to 6(e) are images captured within the field of view of the objective lens 20 (i.e., microscope images), and Figures 6(f) to 6(j) are examples of the display on the three-dimensional display device 32a corresponding to Figures 6(a) to 6(e). The three-dimensional display device 32a displays three-dimensionally a first object 70 that mimics the sample 40, a second object 72 that mimics the holding pipette 16a, and a third object 74 that mimics the injection pipette 16b.

[0059] Figures 6(a) and (b) show the state before the injection pipette 16b is brought close to the sample 40, and the injection pipette 16b is not visible in the captured image. However, the three-dimensional position of the injection pipette 16b can be captured even if it is outside the field of view of the objective lens 20. Therefore, a third object 74, which mimics the injection pipette 16b, is also displayed on the three-dimensional display device 32a in Figures 6(f) and (g). As a result, even if the injection pipette 16b is not visible within the field of view of the objective lens 20, the injection pipette 16b can be manipulated so that its tip approaches the sample 40 while viewing the three-dimensional display device 32a.

[0060] The three-dimensional display device 32a allows the user to arbitrarily switch viewpoints. For example, the display can be switched to a top view when the sample 40 is viewed in the z direction, a side view when the sample 40 is viewed in the x direction, or a side view when the sample 40 is viewed in the y direction. Figure 6(f) shows an example of a top view display when the sample 40 is viewed in the z direction, and Figure 6(g) shows an example of a side view display when the sample 40 is viewed in the y direction. By appropriately switching viewpoints, the insertion angle of the manipulator 16 relative to the sample 40 can also be easily confirmed.

[0061] Figures 6(c) and 6(d) show the state in which the injection pipette 16b is in contact with the sample 40 during manipulation. The image in Figure 6(c) appears to show the injection pipette 16b being inserted into the sample 40. However, the three-dimensional display device 32a in Figure 6(h) shows that the injection pipette 16b is tucked under the sample 40, and is not actually inserted into the sample 40. Furthermore, it can be seen that the sample 40 is being lifted upward by the injection pipette 16b, and that the holding pipette 16a is about to release its hold on the sample 40.

[0062] Figure 6(e) shows the state where the injection pipette 16b is inserted into the sample 40 and the sample 40 is separated from the holding pipette 16a. At this time, by looking at the three-dimensional display device 32a in Figure 6(i), it is possible to check how deep the injection pipette 16b has been inserted into the sample 40. Furthermore, by switching the viewpoint of the three-dimensional display device 32a, it is possible to check whether the injection pipette 16b is inserted near the center of the sample 40.

[0063] Figure 7 is a flowchart showing a three-dimensional position presentation method according to an embodiment. First, the origin of the manipulator 16 is calibrated (S10). The origin of the manipulator 16 is calibrated by imaging the tip 38 of the manipulator 16 at varying working distances, and finding the working distance WD at which the average edge intensity F of the captured images is maximized. max This is performed by identifying the sample. Next, an image of the observation area is acquired (S12), and if the sample 40 is present in the image (Y in S14), multiple images are acquired by changing the working distance WD (S16). The three-dimensional position of the sample 40 is determined based on the maximum value obtained when the average edge intensity F of the sample 40 included in the multiple images is approximated by a Gaussian distribution (S18). Next, the three-dimensional position of the manipulator 16 in the second coordinate system is determined based on the operation of the manipulator 16, and the three-dimensional position of the manipulator 16 in the first coordinate system is determined by a coordinate transformation from the second coordinate system to the first coordinate system (S20). Based on the three-dimensional positions of the sample 40 and the manipulator 16 in the first coordinate system that have been determined, the sample 40 and the manipulator 16 are mapped into a virtual space (S22), and the three-dimensional arrangement relationship of the sample 40 and the manipulator 16 mapped into the virtual space is displayed on a three-dimensional display device 32a or the like (S24). If sample 40 is not found in S14 (N in S14), skip the processes in S16 to S24. If you continue using the manipulation system 10 (N in S26), repeat the processes in S12 to S24. If you stop using the manipulation system 10 (Y in S26), terminate this flow.

[0064] In one example of this disclosure, the processing in S12 to S24 is repeated every 40 milliseconds, and the three-dimensional positions of the sample 40 and manipulator 16 are updated and displayed at a rate of 25 frames per second. Since this display period is almost the same as the frame rate of a typical video, to a user viewing the three-dimensional display device 32a, it appears as if the three-dimensional positions of the sample 40 and manipulator 16 are reflected instantly without any time lag. As a result, the sample 40 can be accurately manipulated with the manipulator 16 while viewing the three-dimensional display device 32a without feeling stressed by the discrepancy between the actual position and the displayed position. Since the three-dimensional display device 32a presents the positional relationship between the sample 40 and the manipulator 16 in a state that allows for stereoscopic viewing, the positional relationship between the sample 40 and the manipulator 16 can be confirmed from various angles by changing the direction of the gaze toward the three-dimensional display device 32a. In other words, since the enlarged sample 40 and manipulator 16 can be displayed as if they were three-dimensionally present in front of the user, it becomes easy to grasp the three-dimensional positional relationship between the two. This improves the operability when precisely manipulating minute samples 40 such as cells, thereby enhancing the convenience of the manipulation system 10.

[0065] Next, we will explain how to estimate the force applied between the sample 40 and the manipulator 16.

[0066] (Force estimation method) Figures 8(a) to 8(c) schematically show how the sample 40 is manipulated by the manipulator 16. Figure 8(a) shows the holding operation of the sample 40, Figure 8(b) shows the rotation operation of the sample 40, and Figure 8(c) shows the puncture operation of the sample 40. Figures 8(a) to 8(c) correspond to the images captured by the imaging device 24 and show the sample 40 as viewed in the z direction. In Figures 8(a) to 8(c), the protruding direction of the tip 38a of the holding pipette 16a is defined as the +x direction.

[0067] Figure 8(a) shows the procedure for holding the sample 40 using a holding pipette 16a. The tip 38a of the holding pipette 16a is brought close to the sample 40, and the sample 40 is fixed by aspirating it with the tip 38a of the holding pipette 16a. The sample 40 is in contact with the tip 38a of the holding pipette 16a in the x-direction, and a holding force Fa in the x-direction (-x direction) due to the aspiration is applied to the sample 40. An example of the sample 40 is a mature human or animal egg cell, which has a central cytoplasm 40a, a zona pellucida 40b surrounding the cytoplasm 40a, and a polar body 40c located in a part between the cytoplasm 40a and the zona pellucida 40b.

[0068] Figure 8(b) shows the operation of rotating the sample 40 using the injection pipette 16b. In Figure 8(b), the position of the polar body 40c is adjusted by rotating the sample 40 as indicated by the arrow R. For example, in the perforation operation in Figure 8(c), the orientation of the sample 40 is adjusted so that the polar body 40c is located in the y direction from the center 40d of the sample 40, in order to prevent the spindle located near the polar body 40c from being damaged by the injection pipette 16b.

[0069] In the rotation operation shown in Figure 8(b), the injection pipette 16b is moved not in the direction of protrusion of its tip 38b (e.g., x-direction), but in a direction perpendicular to the protrusion direction (e.g., y-direction). By pressing the side of the injection pipette 16b against the sample 40, the position and orientation of the sample 40 are shifted. For example, when the injection pipette 16b is pressed against the sample 40 in the +y direction, a force Fb in the +y direction is applied to the sample 40, and the center position of the sample 40 shifts in the +y direction from 40d0 to 40d1. At this time, because the sample 40 tries to return to its original position due to the holding force Fa from the holding pipette 16a, a reaction force Fc is applied to the injection pipette 16b due to the holding force Fa. The direction of this reaction force Fc corresponds to the opposite direction of the displacement d from the center position 40d0 of the sample 40 before the operation to the center position 40d1 of the sample 40 after the operation. Furthermore, the magnitude of the reaction force Fc correlates with the magnitude of the displacement d of the sample 40 caused by the operation of the injection pipette 16b (i.e., the amount of displacement). The reaction force Fc can be expressed, for example, by the following equation (6).

number

number

[0070] The displacement d in equations (6) and (7) above can be determined in real time based on the change in the position of the center 40d of the sample 40. The control device 30 determines the displacement d at each moment by determining the position of the center 40d of the sample 40 using the three-dimensional position determination method based on the captured image described above. Based on the determined displacement d, the control device 30 estimates the force Fc applied between the sample 40 and the injection pipette 16b at each moment.

[0071] Figure 8(c) shows the operation of puncturing the sample 40 with the injection pipette 16b. In Figure 8(c), a force Fd in the -x direction is applied to the sample 40 by pressing the tip 38b of the injection pipette 16b toward the sample 40 in the -x direction. The sample 40 is squeezed between the holding pipette 16a and the injection pipette 16b and deformed, and the area in contact with the tip 38b of the injection pipette 16b becomes concave in the -x direction. At this time, a reaction force Fe is applied to the injection pipette 16b due to the elastic force of the sample 40 trying to return to its original shape. The direction of this reaction force Fe is opposite to the direction of protrusion (-x direction) of the tip 38b of the injection pipette 16b (-x direction). Furthermore, the magnitude of the reaction force Fe correlates with the magnitude of the deformation amount wd of the sample 40, and can be expressed, for example, by the following equation (8).

number

[0072] The deformation amount wd in equation (8) above corresponds to the amount of indentation of the sample 40 in the protruding direction (x direction) of the tip 38b of the injection pipette 16b, and corresponds to the distance in the protruding direction (x direction) from the end 40e of the sample 40 to the tip 38b of the injection pipette 16b. The deformation amount wd can be measured, for example, based on an image of the sample 40. The deformation amount wd may also be specified as the difference between the x-direction distance w1 from the center 40d of the sample 40 to the end 40e and the x-direction distance w2 from the center 40d of the sample 40 to the tip 38b of the injection pipette 16b (i.e., wd = w1 - w2). In this case, the positions of the center 40d and the end 40e of the sample 40 can be specified based on an image of the sample 40. The position of the tip 38b of the injection pipette 16b can be specified by a method utilizing coordinate transformation in the three-dimensional positioning method described above.

[0073] The control device 30 uses the three-dimensional positioning method based on the captured image described above to determine the position of the center 40d of the sample 40 in real time. Based on the captured image that is in the best focus on the sample 40, the control device 30 determines the positions of the center 40d and the end 40e of the sample 40 in real time and determines the distance w1. The control device 30 determines the position of the tip 38b of the injection pipette 16b in real time by performing a coordinate transformation on the three-dimensional position with respect to the drive axis of the injection pipette 16b. As a result, the control device 30 determines the deformation amount wd at each moment. Based on the determined deformation amount wd, the control device 30 estimates the force Fe applied between the sample 40 and the injection pipette 16b at each moment.

[0074] The control device 30 transmits force information based on the estimated forces Fc,Fe to the force presentation device 36, so that a force sensation corresponding to the estimated forces Fc,Fe is presented to the user through the force presentation device 36. Since the magnitude of the estimated forces Fc,Fe is very small, about 1 μN to 1 mN, if the magnitude of the estimated forces Fc,Fe is transmitted directly to the force presentation device 36, it would be difficult for the user to perceive such a small force. Therefore, in this disclosure, the estimated force is amplified and presented to the user. The control device 30 generates force information showing the forces αFc,αFe obtained by multiplying the estimated forces Fc,Fe by a predetermined amplification factor α, and transmits it to the force presentation device 36. The amplification factor α is between 100 and 10,000 times, for example, between 1,000 and 5,000 times. The force presentation device 36 presents the amplified forces αFc,αFe by operating an actuator based on the force information acquired from the control device 30. For example, if the amplification factor α is set to 1000 times, a force sensation of approximately 1 mN to 1 N will be presented, allowing the user to easily perceive the reaction force associated with operating the manipulator 16. The amplification factor α may be a parameter that the user can adjust as appropriate.

[0075] Figure 9 is a flowchart illustrating a force feedback method according to an embodiment. The control device 30 acquires position information from the force feedback device 36 based on user input operations to specify the position of the manipulator 16 (S30). For example, when a user grasps and moves the tip 36b of the articulated arm 36a, position information corresponding to the position of the tip 36b of the articulated arm 36a is transmitted to the control device 30. Subsequently, the control device 30 controls the operation of the manipulator drive mechanism 26 based on the acquired position information to control the position of the manipulator 16 (S32). The imaging device 24 images the sample 40 manipulated using the manipulator 16 through the objective lens 20, and the control device 30 acquires the image captured by the imaging device 24 (S34).

[0076] Next, the control device 30 identifies the amount of change in the position or shape of the sample 40 based on the acquired image (S36), and estimates the force applied between the sample 40 and the manipulator 16 based on the identified amount of change (S38). For example, when the rotation operation shown in Figure 8(b) is performed, the control device 30 identifies the amount of displacement d of the center position of the sample 40, and estimates the direction and magnitude of the reaction force Fc applied to the injection pipette 16b based on the identified displacement d. When the drilling operation shown in Figure 8(c) is performed, the control device 30 identifies the amount of deformation wd of the sample 40, and estimates the direction and magnitude of the reaction force Fe applied to the injection pipette 16b based on the identified deformation wd. The control device 30 transmits force information corresponding to the estimated forces Fc and Fe to the force feedback device 36, and the force feedback is presented to the user through the force feedback device 36 (S40). For example, the control device 30 transmits force information representing forces αFc and αFe, obtained by amplifying the estimated forces Fc and Fe with a predetermined amplification factor α, to the force presentation device 36, so that the amplified forces are presented to the user.

[0077] According to this disclosure, the user can operate the manipulator 16 using the force feedback device 36 while feeling the reaction force applied to the manipulator 16 through the force feedback device 36. Therefore, operation based on both visual and tactile senses becomes possible, and the user can manipulate the sample 40 while understanding what kind of force is applied to the sample 40 depending on the positional relationship between the sample 40 and the manipulator 16. As a result, it is possible to help the user acquire skills to appropriately control the force applied to cells.

[0078] According to this disclosure, force information indicating the magnitude of force sensation to be presented to the user is generated based on an image of the sample 40 being manipulated using the manipulator 16, thereby enabling the presentation of force sensation corresponding to the force applied between the sample 40 and the manipulator 16. Since it is difficult to detect the force applied to the sample 40 by the manipulator 16 in real time using sensors, etc., generating force information based on an image allows for the presentation of force sensation in real time, thereby improving operability.

[0079] Several aspects of this disclosure are described below.

[0080] A first aspect of this disclosure is a manipulation system comprising: a manipulator for manipulating a sample; a manipulator drive mechanism for moving the manipulator; an imaging device for imaging the sample through an objective lens; a control device that identifies the amount of change of at least one of the position and shape of the sample based on the image captured by the imaging device, and estimates the force applied between the sample and the manipulator based on the identified amount of change; and a force feedback device configured to receive input from a user to specify the position of the manipulator and to present a force feedback to the user corresponding to the force estimated by the control device.

[0081] A second aspect of the present disclosure is a manipulation system according to the first aspect, characterized in that the control device identifies the amount of deformation of the sample in the protruding direction of the tip of the manipulator based on the image, and estimates the force based on the amount of deformation.

[0082] A third aspect of the present disclosure is a manipulation system according to the second aspect, characterized in that the control device determines the amount of deformation based on the tip position of the manipulator, the center position of the sample, and the end position of the sample in the protruding direction.

[0083] A fourth aspect of the present disclosure is a manipulation system according to any one of the first to third aspects, characterized in that the control device identifies the amount of displacement of the sample based on the image and estimates the force based on the amount of displacement.

[0084] A fifth aspect of the present disclosure is a manipulation system according to any one of the first to fourth aspects, characterized in that the control device transmits force information indicating a force amplified by 100 times or more the estimated magnitude of the force to the force feedback device, and the force feedback device presents the amplified force to the user.

[0085] A sixth aspect of the present disclosure is a force feedback method characterized by comprising: acquiring position information from a force feedback device based on a user input operation to specify the position of a manipulator; controlling the operation of a manipulator drive mechanism that moves the manipulator based on the acquired position information; imaging a sample manipulated using the manipulator through an objective lens; identifying an amount of change in at least one of the position and shape of the sample based on the image captured; estimating a force applied between the sample and the manipulator based on the identified amount of change; and controlling the operation of the force feedback device so that a force corresponding to the estimated force is presented to the user.

[0086] A seventh aspect of this disclosure is a program that causes a computer to implement the following functions: a function to acquire position information from a force feedback device based on user input operations to specify the position of a manipulator; a function to control the operation of a manipulator drive mechanism that moves the manipulator based on the acquired position information; a function to acquire an image of a sample manipulated using the manipulator captured through an objective lens; a function to identify the amount of change of at least one of the position and shape of the sample based on the acquired image; a function to estimate the force applied between the sample and the manipulator based on the amount of change; and a function to control the operation of the force feedback device so that force corresponding to the estimated force is presented to the user.

[0087] (Second Embodiment) Figure 10 is a schematic diagram showing the configuration of the manipulation system 110 according to the second embodiment. The second embodiment differs from the first embodiment described above in that, in addition to the force feedback device 36 (also called the first force feedback device 36) for operating the injection pipette 16b, a second force feedback device 80 is used for operating the holding pipette 16a. The second embodiment will be described below, focusing on the differences from the first embodiment, and the common points will be omitted as appropriate.

[0088] The manipulation system 110 comprises a stage 12, a lighting device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator drive mechanism 26, a lens drive mechanism 28, a control device 30, a display device 32, an input device 34, a first force feedback device 36, a pump 78, and a second force feedback device 80.

[0089] The first force feedback device 36 is configured in the same manner as the first embodiment described above. The first force feedback device 36 is an input operation device for operating the injection pipette 16b. The articulated arm 36a of the first force feedback device 36 is a first link mechanism for supporting the tip 36b of the articulated arm 36a, which is, for example, a serial link mechanism. The tip 36b of the articulated arm 36a is a first holding member held by the user. Therefore, the first force feedback device 36 has a first holding member held by the user and a first link mechanism that supports the first holding member such that the position of the first holding member is variable according to the user's operation.

[0090] The first force feedback device 36 further includes an actuator 36c for driving the first link mechanism (articulated arm 36a) to apply a reaction force to the first holding member (tip 36b). The first force feedback device 36 further includes a position sensor 36d for detecting the orientation of the articulated arm 36a and detecting the position of the first holding member (tip 36b). The first force feedback device 36 is operated, for example, by the user's right hand.

[0091] Pump 78 is connected to the inside of the holding pipette 16a and generates suction or discharge force at the tip 38a of the holding pipette 16a. Pump 78 is, for example, an electric microinjector. Pump 78 generates suction force at the tip 38a of the holding pipette 16a by creating negative pressure inside the holding pipette 16a. Pump 78 generates discharge force at the tip 38a of the holding pipette 16a by creating positive pressure inside the holding pipette 16a. Pump 78 variably controls the suction and discharge force based on commands transmitted from the control device 30.

[0092] The second force feedback device 80 is an input device for operating the manipulator 16 and an input device for operating the holding pipette 16a. The second force feedback device 80 is configured to accept input operations for specifying the three-dimensional position of the holding pipette 16a and input operations for specifying the suction force and dispensing force at the tip of the holding pipette 16a. The second force feedback device 80 includes a second link mechanism 82, a second holding member 84, and a rotating member 86. The second force feedback device 80 is operated, for example, by the user's left hand.

[0093] The second link mechanism 82 supports the second retaining member 84 such that its position can be varied according to user operation. The second link mechanism 82 is configured such that the second retaining member 84, which is attached to the tip of the second link mechanism 82, can move along the three axes of X, Y, and Z. The second link mechanism 82 is, for example, a parallel link mechanism in which three arms are connected in parallel. The second link mechanism 82 is provided with, for example, an actuator 82a for driving each of the three arms to apply a reaction force to the second retaining member 84, and a position sensor 82b for detecting the orientation of each of the three arms.

[0094] The second holding member 84 is a member held by the user. The second holding member 84 is held, for example, by the user's thumb and middle finger during use. The user performs an input operation to specify the three-dimensional position of the holding pipette 16a by holding and moving the second holding member 84. The three-dimensional position of the second holding member 84 is detected, for example, by a position sensor 82b provided on the second link mechanism 82.

[0095] The rotating member 86 is a component that receives input operations from the user's gripping motion. The rotating member 86 is provided near the second holding member 84 and is configured to be rotatable relative to the second holding member 84. The user performs an input operation to specify the suction force and discharge force of the tip 38a of the holding pipette 16a by rotating the rotating member 86.

[0096] As the second force feedback device 80, for example, an Omega 7 manufactured by Force Dimension can be used, and for example, the active gripper and force feedback device disclosed in International Publication No. 2008 / 003416 can be used.

[0097] Figures 11(a) and 11(b) schematically show the configuration of the second force feedback device 80. Figure 11(a) schematically shows the configuration of the second holding member 84 and the rotating member 86, and Figure 11(b) schematically shows the input operation by the user's gripping motion while holding the second holding member 84.

[0098] The second holding member 84 has a shape that is easy for the user to hold with their fingers, for example, a cylindrical shape. A connecting bar 84a is attached to the second holding member 84, extending radially from the second holding member 84. A pivot shaft 88 is provided at the tip of the connecting bar 84a, which rotatably supports the rotating member 86.

[0099] The rotating member 86 has a linear member 86a, an arc member 86b, and a pad 86c. The linear member 86a is connected to a pivot shaft 88 and is rotatable about the pivot shaft 88. The arc member 86b extends from the end of the linear member 86a toward the pad 86c along an arc centered on the pivot shaft 88. The pad 86c is provided at the end of the arc member 86b and comes into contact with the user's index finger during use.

[0100] The second holding member 84 is provided with a pulley 90a for driving the rotating member 86, an actuator 90b for driving the pulley 90a, and an angle sensor 90c for detecting the rotation angle of the pulley 90. The pulley 90a is configured to mesh with the outer circumference of the arc member 86b. The pulley 90a rotates in proportion to the amount of circumferential movement when the arc member 86b moves in the circumferential direction due to the rotation of the rotating member 86. The actuator 90b drives the pulley 90a, thereby applying a reaction force to the rotating member 86 and providing force feedback to the user operating the rotating member 86. The angle sensor 90c detects the rotation angle θ of the rotating member 86 relative to the second holding member 84 by detecting the rotation angle of the pulley 90a.

[0101] As shown by arrow C1 in Figure 11(b), when a user performs a gripping motion to grasp an object with their fingers, that is, a gripping motion in which the thumb and index finger are brought together opposite each other, the rotating member 86 rotates clockwise relative to the second holding member 84, and the rotation angle θ increases. On the other hand, as shown by arrow C2 in Figure 11(c), when a user performs a gripping motion to release the grasped object, that is, a gripping motion in which the thumb and index finger are spread apart, the rotating member 86 rotates counterclockwise relative to the second holding member 84, and the rotation angle θ decreases.

[0102] The control device 30 acquires position information indicating the three-dimensional position of the second holding member 84 from the second force feedback device 80. Based on the position information acquired from the second force feedback device 80, the control device 30 controls the operation of the first drive mechanism 26a that moves the holding pipette 16a.

[0103] The control device 30 acquires angular information from the second force feedback device 80 indicating the rotation angle θ of the rotating member 86. Based on the angular information acquired from the second force feedback device 80, the control device 30 controls the operation of the pump 78, which variably controls the suction and discharge forces at the tip 38a of the holding pipette 16a. The angular information indicating the rotation angle θ of the rotating member 86 is an example of operation information indicating the control amount that specifies the suction and discharge forces of the pump 78.

[0104] The control device 30 prevents the generation of either suction or discharge force at the tip 38a of the manipulator 16 when the rotation angle θ of the rotating member 86 is a predetermined initial value θ0 (i.e., θ = θ0). In other words, when the rotation angle θ of the rotating member 86 is a predetermined initial value θ0, the operation of the pump 78 is stopped.

[0105] The control device 30 operates the pump 78 so that a suction force is generated at the tip 38a of the manipulator 16 when the rotation angle θ of the rotating member 86 is greater than a predetermined initial value θ0 (i.e., θ > θ0). The control device 30 may operate the pump 78 so that the suction force changes according to the rotation angle θ of the rotating member 86, or it may operate so that the suction force increases as the rotation angle θ of the rotating member 86 increases. For example, the suction force may be made proportional to the magnitude of the difference Δθ (= θ - θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0.

[0106] The control device 30 operates the pump 78 so that a discharge force is generated at the tip 38a of the manipulator 16 when the rotation angle θ of the rotating member 86 is smaller than a predetermined initial value θ0 (i.e., θ < θ0). The control device 30 may also operate the pump 78 so that the discharge force changes according to the rotation angle θ of the rotating member 86, or it may be such that the discharge force increases as the rotation angle θ of the rotating member 86 decreases. For example, the discharge force may be made proportional to the magnitude of the difference Δθ (= θ - θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0.

[0107] The control device 30 generates force information indicating the magnitude and direction of force sensation to be presented to the user through the second force sensation presentation device 80. The control device 30 generates force information so that force sensation is fed back according to the suction force or dispensing force at the tip 38a of the holding pipette 16a.

[0108] The control device 30 generates force information where the magnitude of the force is zero when the rotation angle θ of the rotating member 86 is at a predetermined initial value θ0 (i.e., θ = θ0) and neither suction force nor discharge force is generated. Therefore, when neither suction force nor discharge force is generated, no force feedback is presented to the user. This allows the user to be informed through force feedback whether or not the rotation angle θ of the rotating member 86 is at a predetermined initial value θ0.

[0109] The control device 30 generates force information indicating the magnitude of a force corresponding to the attractive force when the rotation angle θ of the rotating member 86 is greater than a predetermined initial value θ0 (i.e., θ > θ0) and an attractive force is generated. The control device 30 generates force information that is in the opposite direction to the rotation direction of the rotating member 86. As a result, when the rotating member 86 is rotated in the direction of arrow C1 in Figure 11(b), a force sensation is generated in the opposite direction of arrow C2, and a force sensation representing a reaction force can be presented to the user. Furthermore, by increasing the reaction force as the magnitude of the difference Δθ (=θ-θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0 increases, that is, as the attractive force increases, the user can be informed through force sensation that the attractive force is increasing according to the amount of manipulation of the rotating member 86. The control device 30 may also generate force information whose magnitude is proportional to the difference Δθ (=θ-θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0. The control device 30 may generate force information whose magnitude is proportional to the suction force of the pump 78.

[0110] The control device 30 generates force information indicating the magnitude of the force corresponding to the discharge force when the rotation angle θ of the rotating member 86 is smaller than a predetermined initial value θ0 (i.e., θ < θ0) and a discharge force is being generated. The control device 30 generates force information that is in the opposite direction to the rotation direction of the rotating member 86. As a result, when the rotating member 86 is rotated in the direction of arrow C2 in Figure 11(b), a force sensation is generated in the opposite direction of arrow C1, and a force sensation of reaction force can be presented to the user. Furthermore, by increasing the reaction force as the magnitude of the difference Δθ (=θ-θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0 increases, that is, as the suction force increases, the control device 30 can convey to the user through force sensation that the discharge force is increasing according to the amount of manipulation of the rotating member 86. The control device 30 may also generate force information whose magnitude is proportional to the difference Δθ (=θ-θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0. The control device 30 may generate force information whose magnitude is proportional to the discharge force of the pump 78.

[0111] The control device 30 may generate force information in which the magnitude of the reaction force differs depending on whether or not the sample 40 is in contact with the tip 38a of the holding pipette 16a. For example, if the suction force or dispensing force is constant, the reaction force when the sample 40 is in contact with the tip 38a of the holding pipette 16a may be greater than the reaction force when the sample 40 is not in contact with the tip 38a of the holding pipette 16a. In this case, the reaction force increases the moment the sample 40 comes into contact with the tip 38a of the holding pipette 16a, so the user can be informed through force that the sample 40 has come into contact. Also, the reaction force decreases the moment the sample 40 leaves the tip 38a of the holding pipette 16a, so the user can be informed through force that the sample 40 has left.

[0112] Figure 12 is a schematic diagram illustrating the method for determining contact between the sample 40 and the manipulator 16. The control device 30 determines whether the sample 40 is in contact with the tip 38a of the holding pipette 16a based on the image of the sample 40. The control device 30 makes the contact determination based on the position of the sample 40 relative to the position of the tip 38a of the holding pipette 16a. The position of the sample 40 can be determined based on the image of the sample 40. The position of the tip 38a of the holding pipette 16a can be determined by a method utilizing coordinate transformation as described in the three-dimensional position determination method according to the first embodiment.

[0113] The control device 30 determines contact by whether the end portion 40f of the sample 40 is included within the vicinity region 92 of the tip 38a of the holding pipette 16a. The vicinity region 92, shown by the dashed line in Figure 12, is set to a range where, for example, the width is wb in the direction of protrusion of the holding pipette 16a (+x direction), the width is wa in the opposite direction (-x direction), and the width is wc in the direction perpendicular to the protrusion direction (±y direction), with the tip 38a of the holding pipette 16a as the reference point. The size of the vicinity region 92 can be set to be approximately the same as the size of the tip 38a of the holding pipette 16a, and can be set to be less than or equal to the size of the sample 40 (radius rs). For example, if the radius rs of the sample 40 is about 50 μm, then wa = 10 μm, wb = 20 μm, and wc = 10 μm, and the size of the vicinity region 92 is 30 μm × 20 μm.

[0114] The control device 30 identifies the position coordinates (Cx, Cy) of the center 40d of the sample 40 and the radius rs of the sample 40 based on the captured image. The control device 30 identifies the position coordinates of the tip 38a of the holding pipette 16a in the second coordinate system from the operation of the manipulator drive mechanism 26, and identifies the position coordinates (Mx, My) of the tip 38a by performing a coordinate transformation from the second coordinate system to the first coordinate system. The control device 30 calculates the position coordinates (Cx-rs, Cy) of the end 40f of the sample 40 from the position coordinates (Cx, Cy) of the center 40d of the sample 40 and the radius rs of the sample 40. The control device 30 calculates the range of the neighboring region 92 from the position coordinates (Mx, My) of the tip 38a of the holding pipette 16a. The control device 30 determines that the sample 40 is in contact with the tip 38a of the holding pipette 16a if the end 40f of the sample 40 is within the range of the neighboring region 92. Specifically, if Mx-wa ≤ Cx-rs ≤ Mx+wb and My-wc ≤ Cy ≤ My+wc, the control device 30 determines that the sample 40 is in contact with the tip 38a of the holding pipette 16a. If the end portion 40f of the sample 40 is outside the range of the neighboring region 92, the control device 30 determines that the sample 40 is not in contact with the tip 38a of the holding pipette 16a.

[0115] If the control device 30 determines that the sample 40 is not in contact with the tip 38a of the holding pipette 16a, it sets the magnitude of the reaction force to k1Δθ, which is the difference Δθ (=θ-θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0, multiplied by the first coefficient k1. If the control device 30 determines that the sample 40 is not in contact with the tip 38a of the holding pipette 16a, it sets the magnitude of the reaction force to k2Δθ, which is the difference Δθ (=θ-θ0) between the rotation angle θ of the rotating member 86 and the initial value θ0, multiplied by the second coefficient k2. In this case, the second coefficient k2 is greater than the first coefficient k1 (i.e., k2 > k1). The ratio k2 / k1 of the first coefficient k1 to the second coefficient k2 is, for example, 1.1 or more and 5 or less, preferably 1.5 or more and 3 or less, and is 2 as an example. The first coefficient k1 and the second coefficient k2 can be set so that the maximum value of the reaction force applied to the rotating member 86 is 0.5N or more and 5N or less, preferably 1N or more and 2N or less.

[0116] Alternatively, instead of switching the magnitude of the reaction force depending on whether the sample 40 is in contact with the tip 38a of the holding pipette 16a, the magnitude of the reaction force may be continuously changed according to the distance ds from the tip 38a of the holding pipette 16a to the end 40f of the sample 40. For example, if the suction force or dispensing force is constant, the magnitude of the reaction force may decrease as the distance ds from the tip 38a of the holding pipette 16a to the sample 40 increases. For example, the magnitude of the reaction force may be set to f(ds)Δθ using a sigmoid function f(ds) such that the second coefficient k2 is when the distance ds is 0 and the first coefficient k1 is when the distance ds is infinite.

[0117] Next, the procedure for handling the sample 40 using the holding pipette 16a will be explained with reference to Figures 13 to 15. Figures 13 to 15 show the work area 94 for handling the sample 40. The work area 94 is broadly divided into three areas 94a, 94b, and 94c. The first area 94a is where the untreated sample 40 before cell manipulation is placed. The second area 94b is where cell manipulation of the sample 40 is performed using the injection pipette 16b. The third area 94c is where the processed sample 40 after cell manipulation is placed.

[0118] First, as shown in Figure 13, the holding pipette 16a is moved to the first region 94a, and the tip 38a of the holding pipette 16a is brought close to the sample 40 in the first region 94a. The user can move the tip 38a of the holding pipette 16a to the first region 94a by holding and moving the second holding member 84 of the second force feedback device 80.

[0119] Next, near the sample 40, a suction force is generated at the tip 38a of the holding pipette 16a, and the sample 40 is drawn in and held by the tip 38a of the holding pipette 16a. The user generates a suction force at the tip 38a of the holding pipette 16a by gripping the rotating member 86 in a way that increases the rotation angle θ of the rotating member 86. The user can perceive the magnitude of the suction force by the reaction force applied to the rotating member 86. Furthermore, when the sample 40 comes into contact with the tip 38a of the holding pipette 16a and is held by the suction force, the reaction force applied to the rotating member 86 increases. Therefore, the user can confirm whether or not the sample 40 is being held by the tip 38a of the holding pipette 16a by the change in the reaction force applied to the rotating member 86. The user can obtain a sense of force as if they were directly grasping and holding the sample 40 with their hand due to the reaction force applied to the rotating member 86.

[0120] Next, as shown in Figure 14, the holding pipette 16a is moved from the first region 94a to the second region 94b while holding the sample 40. The user can move the sample 40 held at the tip 38a of the holding pipette 16a to the second region 94b by moving the second holding member 84 while maintaining a gripping motion of gripping the rotating member 86. However, the holding of the sample 40 may not be maintained when the holding pipette 16a is moved, and the sample 40 may come off the tip 38a. In this case, the reaction force applied to the rotating member 86 decreases because it is determined that the sample 40 is not in contact, so the user can confirm that the sample 40 has come off the tip 38a of the holding pipette 16a due to the decrease in the reaction force applied to the rotating member 86.

[0121] Next, in the second region 94b, cell manipulation is performed on the sample 40 held at the tip 38a of the holding pipette 16a using the injection pipette 16b. The user can, for example, operate the injection pipette 16b with their right hand via the first force feedback device 36 while operating the holding pipette 16a with their left hand via the second force feedback device 80.

[0122] The user performs cell manipulation using the injection pipette 16b, for example, as shown in Figures 8(b) and (c). In the rotation operation shown in Figure 8(b), the user can adjust the suction force by changing the rotation angle θ of the rotating member 86 with their left hand while moving the injection pipette 16b with their right hand. In the perforation operation shown in Figure 8(c), the user can simultaneously move the injection pipette 16b, adjust the position of the sample 40 by moving the holding pipette 16a, and adjust the suction force of the sample 40. At this time, the user can confirm through force whether the sample 40 is being held by the holding pipette 16a by the change in the reaction force applied to the rotating member 86.

[0123] Next, as shown in Figure 15, the holding pipette 16a is moved from the second region 94b to the third region 94c while holding the sample 40 in which the cell manipulation has been completed. The user can move the sample 40 held at the tip 38a of the holding pipette 16a to the third region 94c by moving the second holding member 84 while maintaining a gripping motion of gripping the rotating member 86.

[0124] Subsequently, in the third region 94c, an ejection force is generated at the tip 38a of the holding pipette 16a. The user can generate an ejection force at the tip 38a of the holding pipette 16a by performing a gripping motion by opening their index finger so that the rotation angle θ of the rotating member 86 becomes smaller. The user can confirm the magnitude of the ejection force by the reaction force applied to the rotating member 86. The sample 40 is released from the tip 38a of the holding pipette 16a by the ejection force, and the holding is released. When the sample 40 is released from the tip 38a of the holding pipette 16a and no longer in contact, the reaction force applied to the rotating member 86 decreases, so the user can confirm that the sample 40 has been released from the tip 38a of the holding pipette 16a by the decrease in the reaction force applied to the rotating member 86.

[0125] If cell manipulation is required for the next sample 40, the holding pipette 16a should be moved from the third region 94c to the first region 94a. When performing cell manipulation on multiple samples 40 sequentially, the procedures shown in Figures 13 to 15 are repeated.

[0126] According to this disclosure, by using the second force feedback device 80, the operation of moving the holding pipette 16a and the operation of aspirating and discharging the sample 40 with the holding pipette 16a can be realized with a single input operating device. For example, as in the first embodiment, when moving the holding pipette 16a using a joystick, it is necessary to operate aspiration and discharging using an operating means other than the joystick, which creates the hassle of switching operating means. If the sample 40 is released when moving the holding pipette 16a while holding the sample 40, it is necessary to switch operating means again in order to hold the sample 40, making the operation complicated. On the other hand, according to this embodiment, since only a single input operating device is required, switching operating means is unnecessary, and operability can be improved.

[0127] According to this disclosure, by using the second force feedback device 80, the suction force and dispensing force at the tip 38a of the holding pipette 16a can be presented to the user as force feedback. The user can adjust the magnitude of the suction force or dispensing force while feeling force feedback corresponding to the magnitude of the force, thereby improving the operability of suction and dispensing.

[0128] According to this disclosure, by determining contact between the tip 38a of the holding pipette 16a and the sample 40, it is possible to provide the user with haptic feedback indicating whether or not the sample 40 is being held by the tip 38a of the holding pipette 16a. As a result, it becomes easier to confirm whether or not the sample 40 is being held by the holding pipette 16a compared to when only visual inspection is used, and the operability when holding or releasing the sample 40 can be improved.

[0129] According to this disclosure, a suction force is generated by a gripping motion in the direction of gripping the rotating member 86, and a reaction force corresponding to the suction force is applied to the rotating member 86, thereby providing a force sensation as if the sample 40 were being directly grasped by hand in order to hold the sample 40. Furthermore, by increasing the reaction force when the sample 40 comes into contact with the tip 38a of the holding pipette 16a and the sample 40 is held, a force sensation that mimics the moment when the sample 40 is directly touched by hand can be provided. When the sample 40 is released from the tip 38a of the holding pipette 16a, the reaction force decreases, so a force sensation that mimics the moment when the sample 40 that was being grasped by hand is released can be provided. As a result, a force sensation that is less unnatural for the user can be provided, and the operability of aspiration and dispensing can be more effectively improved.

[0130] To confirm the effectiveness of the manipulation system 110 according to this disclosure, a demonstration experiment was conducted to measure the working time required for a series of operations shown in Figures 13 to 15. In the demonstration experiment, microbeads were used as the sample 40, and the time taken to move three microbeads from the first region 94a to the third region 93 was measured. In the comparative example, the joystick shown in the configuration of the first embodiment was used as the input device for the holding pipette 16a. In Example 1, the second force feedback device 80 was used as the input device, and force feedback was not provided through the rotating member 86. In Example 2, the second force feedback device 80 was used as the input device, and force feedback was provided through the rotating member 86.

[0131] For each of the Comparative Example, Example 1, and Example 2, the average time required by six adults with no prior experience in cell manipulation was calculated. The average time was 116 seconds for the Comparative Example, 51 seconds for Example 1, and 42 seconds for Example 2. The p-values ​​of Welch's t-test were less than 0.01 between the Comparative Example and Example 1, and between Example 1 and Example 2. This indicates that using the second force feedback device 80 as an input device significantly improves operability. Furthermore, it was found that applying a reaction force to the rotating member 86 to provide force feedback when using the second force feedback device 80 as an input device significantly improves operability.

[0132] Figure 16 is a flowchart illustrating an example of a manipulation method according to this disclosure. The control device 30 acquires position information from at least one of the first force feedback device 36 and the second force feedback device 80 based on user input operations to specify the position of the manipulator 16 (S50). Based on the acquired position information, the control device 30 controls the operation of the manipulator drive mechanism 26 to move the manipulator 16 (S52). The imaging device 24 images the sample 40 manipulated using the manipulator 16 through the objective lens 20 (S54). Based on the image captured by the imaging device 24, the control device 30 generates force information indicating the magnitude of the force sensation to be presented to the user (S56). The control device 30 controls the operation of at least one of the first force feedback device 36 and the second force feedback device 80 so that a force sensation corresponding to the generated force information is presented to the user (S58).

[0133] Figure 17 is a flowchart illustrating an example of a manipulation method according to this disclosure. The control device 30 acquires position information indicating the position of the second holding member 84 from the second force feedback device 80 (S70). The control device 30 acquires angle information indicating the rotation angle of the rotating member 86 from the second force feedback device 80 (S72). Based on the acquired position information, the control device 30 controls the operation of the manipulator drive mechanism 26 to move the manipulator 16 (S74). Based on the acquired angle information, the control device 30 controls the operation of the pump 78 to control the suction force and discharge force at the tip of the manipulator 16 (S76). The imaging device 24 images the sample 40 being manipulated using the manipulator 16 through the objective lens 20 (S78). Based on the image captured by the imaging device 24, the control device 30 determines whether or not the sample 40 and the manipulator 16 are in contact (S80).

[0134] If the sample 40 and the manipulator 16 are in contact (Y in S82), the control device 30 generates force information indicating the magnitude of the force, which is the value obtained by multiplying the rotation angle (θ-θ0) by a second coefficient k2 (S84). If the sample 40 and the manipulator 16 are not in contact (N in S82), the control device 30 generates force information indicating the magnitude of the force, which is the value obtained by multiplying the rotation angle (θ-θ0) by a first coefficient k1 (S86). Here, the second coefficient k2 may be greater than the first coefficient k1 (i.e., k2 > k1). The control device 30 controls the operation of the second force feedback device 80 so that force feedback according to the generated force information is presented to the user through the rotating member 86 (S88).

[0135] In the flow chart of Figure 17, the processing steps S80-S88, which provide force feedback to the user, may be omitted. Even if force feedback is not provided to the user, user operability can be improved, as demonstrated in the above-described embodiment 1.

[0136] (Third embodiment) Figure 18 is a schematic diagram showing the configuration of the manipulation system 210 according to the third embodiment. The third embodiment differs from the second embodiment described above in that it further uses a third force feedback device 100 to present to the user the force feedback received by the sample 40 through the operation of the manipulator 16. The third embodiment will be described below, focusing on the differences from the second embodiment, and the common points will be omitted from the explanation as appropriate.

[0137] The manipulation system 210 comprises a stage 12, a lighting device 14, a manipulator 16, a folding mirror 18, an objective lens 20, a variable focus lens 22, an imaging device 24, a manipulator drive mechanism 26, a lens drive mechanism 28, a control device 30, a display device 32, an input device 34, a first force feedback device 36, a pump 78, a second force feedback device 80, and a third force feedback device 100.

[0138] The third force feedback device 100 is attached to the user's body and is configured to present a planar force sensation on the user's body surface. The third force feedback device 100 is, for example, attached to the user's forearm 108 and is configured to apply a force that constricts the forearm. The third force feedback device 100 presents force sensations corresponding to the deformation of the sample 40 manipulated by the manipulator 16. The third force feedback device 100 simulates and presents to the user the force sensations that the sample 40 would be experiencing as a result of the manipulation by the manipulator 16. By using the third force feedback device 100, the user can experience the change in shape of the manipulated sample 40 as a constricting force sensation on the user. This makes it easier to grasp the extensibility of the sample 40.

[0139] Here, the extensibility of sample 40 refers to the degree of deformation of sample 40 during the perforation operation shown in Figure 8(c) above, from the time the injection pipette 16b penetrates the transparent zona 40b of sample 40 until perforation occurs. High extensibility of sample 40 corresponds to a case where the transparent zona 40b is relatively flexible, and perforation occurs after sample 40 has deformed significantly into a concave shape. Conversely, low extensibility of sample 40 corresponds to a case where perforation occurs without much deformation of sample 40.

[0140] It is known that if sample 40 has low extensibility, there is a higher possibility of degeneration occurring in sample 40 after cell manipulation. Therefore, it is desirable to confirm the extensibility of sample 40 during cell manipulation and to be able to ascertain the integrity of sample 40. On the other hand, if sample 40 is perforated multiple times in order to confirm its extensibility, it will damage sample 40. Therefore, it is necessary to be able to reliably ascertain the extensibility of sample 40 in a single perforation. In this embodiment, the degree of deformation of sample 40 is presented to the user as force feedback to support the assessment of the extensibility of sample 40.

[0141] Figure 19 is a schematic diagram showing the configuration of the third force feedback device 100. The third force feedback device 100 comprises a fixing device 102, a plurality of actuators 104a, 104b, 104c, 104d, 104e, 104f, and 104g, and a drive control device 106.

[0142] The fixing device 102 is a component for fixing multiple actuators 104a to 104g to the user's body surface. When attached to the user's forearm 108, an example of the fixing device 102 is an arm cover made of a cylindrical, stretchable fabric. The fixing device 102 may also have a band or the like for wrapping around and securing it to the user's body surface.

[0143] The multiple actuators 104a to 104g are, for example, McKibben-type artificial muscles and have elastic members such as rubber tubes that are driven by air pressure. The multiple actuators 104a to 104g are attached to the cylindrical brace 102 by being wrapped around its outer circumference. The multiple actuators 104a to 104g apply a force that tightens the brace 102 by increasing the air pressure. The multiple actuators 104a to 104g are arranged at intervals along the axial direction L of the cylindrical brace 102 and apply force to multiple different points along the axial direction L. In the illustrated example, seven actuators 104a to 104g are arranged at 30mm intervals along the axial direction L, but the number and spacing of actuators are not particularly limited.

[0144] The drive control device 106 drives a plurality of actuators 104a to 104g. The drive control device 106 includes, for example, a compressor for generating compressed air, a tank for storing compressed air, and a plurality of electro-pneumatic regulators for controlling the air pressure supplied to each of the plurality of actuators 104a to 104g. By changing the air pressure supplied to each of the plurality of actuators 104a to 104g, the drive control device 106 variably controls the magnitude of the force that the plurality of actuators 104a to 104g apply to multiple points on the user's body surface. The drive control device 106 independently variably controls the magnitude of the force applied by each of the plurality of actuators 104a to 104g. This makes it possible to present force sensations with different magnitudes at multiple points on the user's body surface.

[0145] Returning to Figure 18, the control device 30 generates multiple force information to control the operation of the third force feedback device 100. The control device 30 generates multiple force information to drive multiple actuators 104a to 104g. Based on the captured image of the sample 40, the control device 30 identifies the amount of deformation of the sample 40 at multiple locations. Based on the amount of deformation of the sample 40 at the identified multiple locations, the control device 30 generates multiple force information corresponding to the amount of deformation at each of the multiple locations.

[0146] Figure 20 schematically shows the deformation amounts Da to Dg at multiple positions of the sample 40 manipulated by the manipulator 16. Figure 20 shows the puncture operation on the sample 40, with the outline of the sample 40 before the puncture operation begins shown by a dashed line. The sample 40 is deformed by being crushed in the x-direction due to the force F applied by the puncture operation in the x-direction by the injection pipette 16b.

[0147] The control device 30 identifies the deformation amounts Da, Db, Dc, Dd, De, Df, and Dg of the sample 40 at multiple different positions in the y-direction perpendicular to the drilling direction (-x-direction) based on the captured image. The control device 30 generates multiple force information corresponding to the identified deformation amounts Da to Dg. For example, the control device 30 generates multiple force information whose magnitude is proportional to the identified deformation amounts Da to Dg. In this case, the larger the identified deformation amounts Da to Dg, the larger the force information generated. By operating the third force feedback device 100 based on such force information, a planar force sensation can be presented as if the user's forearm 108 is being crushed in the same shape as the sample 40. For example, if the third force feedback device 100 is attached to the user's left arm and the injection pipette 16b is operated with the user's right hand, the user can feel the degree of deformation of the sample 40 with their left arm while operating the sample 40 with their right hand.

[0148] Figure 21 schematically shows a method for calculating the deformation amount of sample 40 using optical flow. The control device 30 can calculate the deformation amounts Da to Dg of sample 40 in real time by calculating the optical flow of the captured image. The control device 30 divides the range 112 containing the sample 40 to be drilled in the captured image into multiple micro-regions 114, and calculates the velocity vector 116 in each micro-region 114 using a known optical flow method such as the Lucas-Kanade method. The velocity vector 116 indicates the direction and magnitude of the image change between the captured image at time t-1 and the captured image at time t. The number of divisions in the y direction of the multiple micro-regions 114 is set to 7, corresponding to the number of actuators 104a to 104g. The number of divisions n in the x direction of the multiple micro-regions 114 is not particularly limited, but can be set to the same number of divisions as in the y direction. The velocity vector 116 has components in the x direction and components in the y direction.

[0149] The control device 30 calculates the deformation amount δD of the sample 40 in the x-direction at a specific y-direction position by summing the x-components of the velocity vector 116 over the range of x1 to xn for each y-direction position. For example, at the third y-direction position yc from the top in Figure 21, the deformation amount δDc of the sample 40 in the x-direction at position yc is calculated by adding all the x-components of the velocity vector 116 in the range of x1 to xn enclosed by the rectangular frame 118. The control device 30 calculates the deformation amount δDi (i=a to g) at each of the multiple y-positions yi (i=a to g) by summing the x-components of the velocity vector 116. The deformation amount δDi represents the minute deformation amount of the sample 40 between time t-1 and time t.

[0150] The control device 30 calculates the deformation amount δDi in the x-direction at multiple y-positions yi each time an image is acquired. The control device 30 calculates the deformation amount Di(i=a~g) of the sample 40 in the x-direction from the start time t=0 to the current time t by integrating the deformation amount δDi calculated from the start time t=0 to the current time t. In this way, the control device 30 can calculate the deformation amount Di(i=a~g) of the sample 40 in the x-direction using the optical flow of the acquired image.

[0151] The control device 30 generates multiple force information based on the deformation amount Di of the sample 40 in the x-direction at multiple y-positions yi. For example, the control device 30 calculates the air pressure Pi (i=a~g) for driving multiple actuators 104a~104g as multiple force information. The air pressure Pi can be made proportional to the deformation amount Di, and can be expressed as, for example, Pi=βDi using a proportionality constant β. The control device 30 may set the air pressure Pi=0 if the deformation amount Di is negative, that is, if the size of the sample 40 is calculated to be larger than the initial state. The control device 30 may set the air pressure Pi to be equal to a predetermined upper limit P0 if the deformation amount Di exceeds a predetermined upper limit.

[0152] Next, we will explain the procedure for perforating the sample 40 using the manipulation system 210. Figures 22(a) to (c) schematically show the perforation procedure of the sample 40 using the manipulator 16.

[0153] Figure 22(a) shows the sample 40 being held by the holding pipette 16a, and represents the state before the start of the perforation operation with the injection pipette 16b. Figure 22(a) shows the state in which the sample 40 has not been deformed by the operation of the injection pipette 16b, and corresponds to the start time t=0 of the perforation operation of the sample 40. The control device 30 acquires an image as shown in Figure 22(a) and starts calculating the optical flow of the sample 40.

[0154] Figure 22(b) shows the state of the sample 40 being perforated by the injection pipette 16b, and shows the state of the sample 40 immediately before perforation. The sample 40 is deformed as if being crushed in the x direction due to the force F in the x direction applied by the tip 38b of the injection pipette 16b. The size of the sample 40 in the x direction decreases from w3 before the perforation operation to w4 during the perforation operation. The control device 30 calculates the deformation amount Di at multiple positions of the sample 40 from the optical flow of the sample 40 calculated based on the captured image during the perforation operation, and calculates the air pressure Pi = βDi corresponding to the deformation amount Di. The third force feedback device 100 drives multiple actuators 104a to 104g based on the air pressure Pi calculated by the control device 30, and presents planar force feedback to the user.

[0155] Figure 22(c) shows the state of the sample 40 immediately after perforation with the injection pipette 16b, where the tip 38b of the injection pipette 16b has reached the inside of the sample 40. When the sample 40 is perforated and the zona pellucida 40b is ruptured, the zona pellucida 40b of the sample 40 is no longer pressed by the tip 38b of the injection pipette 16b, and the sample 40 tries to return to its original size. As a result, the deformation amount Di calculated by the control device 30 becomes smaller than the state immediately before perforation in Figure 22(b), and the air pressure Pi=βDi calculated by the control device 30 also becomes smaller. The third force feedback device 100 presents the user with a planar force feedback that shows a smaller force magnitude compared to the state immediately before perforation in Figure 22(b). Through the third force feedback device 100, the user can feel the decrease in tightness during the perforation operation of the sample 40, and thus understand that perforation has occurred in the sample 40 through force feedback.

[0156] The control device 30 may evaluate the extensibility of the sample 40 based on the captured image. The extensibility CE of the sample 40 can be expressed as CE = (w3 - w4) / w3, where w3 is the x-direction size of the sample 40 before the drilling operation and w4 is the x-direction size of the sample 40 immediately before drilling. w3 - w4 corresponds to the amount of deformation D of the sample 40 from before the drilling operation until immediately before drilling. The extensibility CE can also be called the deformation rate, which is the ratio of the amount of deformation D based on the size of the sample 40.

[0157] The control device 30 can determine the sizes w3 and w4 of the sample 40 using the captured image. The x-direction size w4 of the sample 40 immediately before perforation may be calculated from the coordinates of the tip 38a of the holding pipette 16a and the tip 38b of the injection pipette 16b, as the distance between these coordinates. Based on the captured image, the control device 30 can determine the moment of perforation by detecting the reversal of the deformation direction of the sample 40 when a force F is applied to the sample 40 from the injection pipette 16b in the perforation direction (-x direction). For example, the control device 30 identifies the moment of perforation as the timing when the deformation amount δDi of the sample 40 in the x direction, calculated based on optical flow, changes from the -x direction to the +x direction.

[0158] The control device 30 may determine whether the extensibility is good or bad based on whether the calculated extensibility CE exceeds a predetermined threshold, and output the determination result. The control device 30 may output a determination result indicating that the extensibility is good if the calculated extensibility CE exceeds a predetermined threshold (e.g., 0.7). The control device 30 may output a determination result indicating that the extensibility is poor if the calculated extensibility CE is below a predetermined threshold (e.g., 0.7). The control device 30 may also output the calculated numerical value of the extensibility CE. In this case, since the extensibility CE of the sample 40 is automatically evaluated by the control device 30, even a novice with little experience in cell manipulation can appropriately evaluate the extensibility CE of the sample 40 during the perforation operation.

[0159] The present disclosure has been described above based on embodiments. Those skilled in the art will understand that the present disclosure is not limited to the above embodiments, that various design changes are possible, and that various modifications are possible, and that such modifications are also within the scope of the present disclosure. [Industrial applicability]

[0160] According to this disclosure, it is possible to improve the operability when manipulating a sample using a manipulator. [Explanation of symbols]

[0161] 10, 110, 210... Manipulation system, 12... Stage, 16... Manipulator, 16a... Holding pipette, 16b... Injection pipette, 20... Objective lens, 22... Variable focus lens, 24... Imaging device, 26... Manipulator drive mechanism, 28... Lens drive mechanism, 30... Control device, 32... Display device, 34... Input device, 36... Force feedback device, 38a, 38b... Tip, 40... Sample, 78... Pump, 80... Second force feedback device, 84... Second holding member, 86... Rotating member, 100... Third force feedback device, 102... Fixation device, 104a~104g... Actuator, 106... Drive control device.

Claims

1. A manipulator for handling the sample, A manipulator drive mechanism for moving the aforementioned manipulator, A pump that variably controls the suction and discharge forces at the tip of the manipulator, An input operation device including a holding member held by the user, a link mechanism that supports the holding member such that its position is variable according to the user's operation, a position sensor for detecting the position of the holding member, a rotating member that rotates relative to the holding member by the user's gripping motion, and an angle sensor for detecting the rotation angle of the rotating member, A manipulation system comprising: a control device that controls the operation of the manipulator drive mechanism based on position information indicating the position of the holding member detected by the position sensor; and a control device that controls the operation of the pump based on angle information indicating the rotation angle of the rotating member detected by the angle sensor.

2. The control device stops the suction and discharge by the pump when the rotation angle is at a predetermined initial value. The control device generates a suction force in the pump when the rotation angle changes from the initial value due to a gripping motion that brings the user's thumb and index finger closer together. The control device generates a discharge force in the pump when the rotation angle changes from the initial value due to a gripping motion that moves the user's thumb and index finger further apart. The manipulation system according to claim 1.

3. The control device increases the suction force of the pump or decreases the discharge force of the pump when the rotation angle changes due to a gripping motion that brings the user's thumb and index finger closer together. The control device reduces the suction force of the pump or increases the discharge force of the pump when the rotation angle changes due to a gripping motion that moves the user's thumb and index finger apart. The manipulation system according to claim 1.

4. The control device generates force information corresponding to the magnitude of the suction or discharge force of the pump, The input operating device further includes an actuator that applies a reaction force to the rotating member according to the force information. The manipulation system according to any one of claims 1 to 3.

5. A step of acquiring position information indicating the position of a holding member held by a user, which is supported by a link mechanism such that its position is variable in response to the user's operation; The steps include acquiring angular information indicating the rotation angle of a rotating member that rotates relative to the holding member due to the user's gripping motion, Based on the acquired position information, the operation of the manipulator drive mechanism that moves the manipulator for manipulating the sample is controlled. A manipulation method comprising the steps of controlling the operation of a pump that variably controls the suction force and discharge force at the tip of the manipulator based on the angle information acquired.

6. A function to acquire positional information indicating the position of a holding member held by a user, which is supported by a link mechanism such that its position is variable in response to the user's operation, A function to acquire angular information indicating the rotation angle of a rotating member that rotates relative to the holding member due to the user's gripping action, Based on the acquired position information, the function controls the operation of the manipulator drive mechanism that moves the manipulator for manipulating the sample, A program characterized by causing a computer to implement a function that controls the operation of a pump that variably controls the suction force and discharge force at the tip of the manipulator based on the angle information acquired.

Images