ENDOSCOPYRIGHT: 20100236644

Abstract

This endoscope system comprises an endoscope, a moving device that holds and moves the endoscope, and one or more processors, wherein the one or more processors detect the moving direction (Vtip) of the endoscope moved by the moving device in a coordinate system (Σr) of the moving device, estimate the region of a movable body (7) in an image (C) captured by the endoscope, detect the moving direction (Vobj) of an object in the image on the basis of the other region of the image excluding the region of the movable body (7), calculate a displacement (θ) between the coordinate system (Σe) of the endoscope and the coordinate system (Σr) of the moving device on the basis of the moving direction (Vtip) of the endoscope and the moving direction (Vobj) of the object, and correct the coordinate system (Σr) of the moving device on the basis of the displacement (θ).

Description

[Technical field]

[0001] The present invention relates to an endoscope system and a coordinate system correction method. [Background technology]

[0002] Conventionally, there is known an endoscope system in which an endoscope is moved by a moving device such as an electric holder. In order to move the endoscope accurately, it is desirable that the coordinate system of the endoscope and the coordinate system of the moving device coincide with each other. For example, when an operator remotely controls an electric holder using a user interface, the operator inputs a desired movement direction of the endoscope to the user interface based on an endoscopic image. If the coordinate system of the endoscope and the coordinate system of the moving device match, the user can intuitively and accurately move the endoscope in the desired direction. On the other hand, if the coordinate system of the endoscope and the coordinate system of the moving device do not match, the actual movement direction of the endoscope differs from the movement direction input to the user interface, making it difficult for the user to intuitively and accurately move the endoscope in the desired direction.

[0003] Meanwhile, in a robot system in which a camera system is attached to a movable arm, a method for correcting the transformation between the robot's coordinate system and the coordinate system of the camera system is known (see, for example, Patent Document 1). In Patent Document 1, an image of a target is acquired by the camera system, and the transformation is determined from the position of the movable arm when the image was acquired and the positions of feature points of the target in the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,188,973 Summary of the Invention [Problem to be solved by the invention]

[0005] In the case of Patent Document 1, the conversion between the coordinate systems can be determined only under the condition that the position of the target, which is the subject, is fixed. In other words, Patent Document 1 does not take into consideration a moving subject, and when the subject in the image moves, the conversion between the robot's coordinate system and the camera system's coordinate system cannot be accurately corrected. On the other hand, endoscopic images often contain moving objects, so when the method of Patent Document 1 is applied to an endoscope system, it is difficult to accurately correct the coordinate system.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an endoscopic system and a coordinate system correction method that can accurately correct the coordinate system even when a moving subject is present in the image. [Means for solving the problem]

[0007] One aspect of the present invention is an endoscopic system comprising an endoscope, a moving device that holds and moves the endoscope, and one or more processors, in which the one or more processors detect the movement direction of the endoscope moved by the moving device in the coordinate system of the moving device, estimate a region of a moving object in an image captured by the endoscope, detect the movement direction of the subject in the image based on other regions of the image excluding the region of the moving object, calculate a deviation between the coordinate system of the endoscope and the coordinate system of the moving device based on the movement direction of the endoscope and the movement direction of the subject, and correct the coordinate system of the moving device based on the calculated deviation.

[0008] Another aspect of the present invention is an endoscopic system comprising an endoscope, a moving device that holds and moves the endoscope, and one or more processors, wherein the one or more processors detect the movement direction of the endoscope moved by the moving device in the coordinate system of the moving device, estimate a movement vector of each position in an image captured by the endoscope, detect the movement direction of a subject in the image based on other areas of the image excluding areas where the magnitude of the movement vector is equal to or greater than a predetermined threshold, calculate a deviation between the coordinate system of the endoscope and the coordinate system of the moving device based on the movement direction of the endoscope and the movement direction of the subject, and correct the coordinate system of the moving device based on the calculated deviation.

[0009] Another aspect of the present invention is , move Moved by a moving device did Detecting the movement direction of the endoscope in the coordinate system of the moving device death A region of a moving object in an image captured by the endoscope is estimated. death Detecting the direction of movement of the subject in the image based on the other areas of the image excluding the area of ​​the moving object. death Calculating a deviation between the coordinate system of the endoscope and the coordinate system of the moving device based on the moving direction of the endoscope and the moving direction of the subject death and correcting a coordinate system of the moving device based on the calculated deviation. METHOD FOR OPERATION OF ENDOSCOPIC SYSTEM It is. Effect of the Invention

[0010] According to the present invention, it is possible to accurately correct the coordinate system even when a moving subject is present in an image. [Brief description of the drawings]

[0011] [Figure 1] 1 is an overall configuration diagram of an endoscope system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the overall configuration of the endoscope system of FIG. 1. [Figure 3A]FIG. 4 is a diagram showing an example of an endoscopic image displayed on the display device, and is a diagram illustrating the relationship between an endoscope coordinate system and a holder coordinate system. [Figure 3B] FIG. 11 is a diagram showing an example of an endoscopic image displayed on the display device, and is a diagram illustrating the relationship between a rotated endoscope coordinate system and a holder coordinate system. [Figure 3C] FIG. 13 is a diagram showing an example of an endoscopic image displayed on the display device after the endoscope has been moved, and is a diagram illustrating the relationship between a rotated endoscope coordinate system and a holder coordinate system. [Figure 4A] 1A and 1B are diagrams showing an example of a treatment tool region and other regions in an endoscopic image. [Figure 4B] 1A to 1C are diagrams showing an example of a treatment tool region, margins, and other regions in an endoscopic image. [Diagram 5] 4 is a flowchart of a coordinate system correction method according to the first embodiment of the present invention. [Figure 6] FIG. 11 is a diagram showing an oblique-viewing endoscope in an endoscope system according to a second embodiment of the present invention. [Figure 7] 11A and 11B are diagrams for explaining a method of calculating an endoscope coordinate system for an oblique-viewing endoscope. [Figure 8] FIG. 11 is a diagram for explaining a method for estimating a movement vector of a subject by an endoscope system according to a third embodiment of the present invention, and is a diagram showing the movement vector of each position within an endoscopic image. [Figure 9] FIG. 13 is a diagram for explaining a method for estimating a movement vector of a subject by an endoscope system according to a fourth embodiment of the present invention, and is a table showing the directions and numbers of movement vectors. [Figure 10] FIG. 13 is a diagram for explaining a method for estimating a motion vector of a subject by an endoscope system according to a fifth embodiment of the present invention, and is a diagram showing the motion vectors of each position within an endoscopic image that has been subjected to enhancement processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (First embodiment) An endoscope system and a coordinate system correction method according to a first embodiment of the present invention will be described with reference to the drawings. As shown in Figures 1 and 2, the endoscopic system 1 of this embodiment includes an endoscope 2, a moving device 3 that holds and moves the endoscope 2, an operation device 4 that is operated by a user, a control device 5 that controls the moving device 3 based on an operation signal from the operation device 4, and a display device 6.

[0013] The endoscope 2 is a rigid endoscope having a long and rigid lens barrel 2a, an optical system 2b arranged in the lens barrel 2a for collecting light from a subject, and an image sensor 2c fixed to the lens barrel 2a for capturing an image of the collected light. The endoscope 2 is a direct-view endoscope having a visual axis (optical axis) B coaxial with a longitudinal axis A of the lens barrel 2a. The image sensor 2c is an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0014] The endoscope 2 is inserted into the body together with one or more treatment tools 7, acquires an endoscopic image C (see FIGS. 3A to 3C) including the one or more treatment tools 7 by the imaging element 2c, and transmits the endoscopic image C to the display device 6. The display device 6 is any display such as a liquid crystal display, an organic EL display, or the like. The surgeon operates the treatment tool 7 while observing the endoscopic image C displayed on the display device 6.

[0015] The moving device 3 includes an electric holder 3a that three-dimensionally controls the position and posture of the endoscope 2. The electric holder 3a is a robot arm having multiple joints 3b, and the base end of the endoscope 2 is held at the tip of the electric holder 3a so as to be rotatable around the longitudinal axis A. The tip of the electric holder 3a and the endoscope 2 move integrally by the operation of the multiple joints 3b, and the position and posture of the endoscope 2 are changed three-dimensionally. The moving device 3 includes an angle sensor 3c that detects the rotation angle of each of the multiple joints 3b. The angle sensor 3c is, for example, an encoder, potentiometer, or Hall sensor provided at each joint 3b.

[0016] The operation device 4 has a user interface 4a including input devices such as keys, a joystick, buttons, and a touch panel. The operator can input instructions for moving the endoscope 2 to the operation device 4 by operating the user interface 4a. The operation device 4 transmits an operation signal based on the operation of the user interface 4a to the control device 5. The user interface 4a can also receive a trigger input from the user. The trigger is for executing a process for correcting the holder coordinate system Σr, as will be described later.

[0017] As shown in FIG. 2, the control device 5 includes at least one processor 5a, a memory 5b, a storage unit 5c, an input interface 5d, and an output interface 5e. The control device 5 is connected to other surrounding devices 2, 3, 4, 6 via an input interface 5d and an output interface 5e, and transmits and receives an endoscopic image C, information on the rotation angle of the joint 3b, signals, etc. via the interfaces 5d, 5e.

[0018] The memory 5b is, for example, a semiconductor memory including a read-only memory (ROM) or a random access memory (RAM) area. The storage unit 5c is a non-transitory recording medium readable by a computer, and is, for example, a non-volatile recording medium including a hard disk or a semiconductor memory such as a flash memory. The processor 5a controls the movement device 3 based on an operation signal from the operation device 4, thereby moving the endoscope 2 according to an instruction input by the operator to the user interface 4a.

[0019] Here, as shown in Fig. 3A, the holder coordinate system Σr is initially set so that the holder coordinate system Σr and the endoscope coordinate system Σe coincide with each other. The endoscope coordinate system Σe is a coordinate system fixed with respect to the lens barrel portion 2a and the image sensor 2c, and the holder coordinate system Σr is a coordinate system fixed with respect to the tip of the electric holder 3a.

[0020] In one example, the endoscope coordinate system Σe is an orthogonal coordinate system having mutually orthogonal Xe, Ye, and Ze axes, and the holder coordinate system Σr is an orthogonal coordinate system having mutually orthogonal Xr, Yr, and Zr axes. The Xe and Xr axes coincide with the longitudinal axis A, the Ye and Yr axes are parallel to the horizontal direction (left-right direction) of the endoscope image C, and the Ze and Zr axes are parallel to the vertical direction (up-down direction) of the endoscope image C. The holder coordinate system Σr and the endoscope coordinate system Σe coincide with each other means that the directions of the Xe and Xr axes coincide with each other, the directions of the Ye and Yr axes coincide with each other, and the directions of the Ze and Zr axes coincide with each other.

[0021] 3B, however, a deviation occurs between the endoscope coordinate system Σe and the holder coordinate system Σr due to the endoscope 2 being rotated about the longitudinal axis A relative to the moving device 3. That is, the endoscope coordinate system Σe rotates about the Xe axis corresponding to the longitudinal axis A, and the endoscope coordinate system Σe is shifted relative to the holder coordinate system Σr by the rotation angle θ of the endoscope 2 in the rotation direction about the Xe axis.

[0022] When the endoscope coordinate system Σe does not coincide with the holder coordinate system Σr in this manner, it is difficult for a user observing the endoscopic image C displayed on the display device 6 to intuitively and accurately move the endoscope 2 in the desired direction by operating the user interface 4a. For example, if the endoscope coordinate system Σe is rotated 90° relative to the holder coordinate system Σr, when the user inputs an instruction to move the endoscope 2 upward in the endoscopic image C into the user interface 4a, the movement device 3 moves the endoscope 2 to the left in the endoscopic image C.

[0023] The processor 5a can execute a process of correcting the holder coordinate system Σr based on the deviation between the coordinate systems Σe and Σr. Specifically, as shown in FIG. 2, the processor 5a includes a treatment tool area estimation unit (moving object area estimation unit) 51 that estimates a treatment tool area (moving object area) D in the endoscopic image C, a subject movement detection unit 52 that detects the movement direction of the subject S in the endoscopic image C, a holder movement detection unit 53 that detects the movement direction of the tip of the electric holder 3a, a deviation calculation unit 54 that calculates the deviation between coordinate systems Σe, Σr based on the movement direction of the subject S and the movement direction of the tip of the electric holder 3a, a coordinate system correction unit 55 that corrects the holder coordinate system Σr based on the estimated deviation, and a control unit 56 that controls the moving device 3 based on the holder coordinate system Σr.

[0024] The processor 5a executes processing in accordance with a coordinate system correction program (not shown) recorded in the storage unit 5c and read into the memory 5b, thereby realizing the functions of the units 51, 52, 53, 54, 55, and 56, which will be described later.

[0025] As shown in FIG. 4A, the treatment tool area estimation unit 51 recognizes a treatment tool (moving object) 7 in an endoscopic image C using a known method such as image recognition using artificial intelligence, and estimates the area of ​​the treatment tool 7 as a treatment tool area D. The subject movement detection unit 52 detects the movement direction of the subject S based on other regions E of the endoscopic image C excluding the treatment tool region D. Specifically, the subject movement detection unit 52 uses a known method such as optical flow or visual SLAM to estimate a movement vector Vobj of the subject S in other regions E from two or more endoscopic images C captured at different times. The movement vector Vobj is a two-dimensional vector representing the movement direction of the subject S in the YeZe plane perpendicular to the Xe axis, and represents the movement direction of the endoscope 2 in the endoscope coordinate system Σe. Specifically, the movement vector Vobj and the movement direction of the endoscope 2 are opposite to each other.

[0026] The holder movement detection unit 53 acquires the rotation angle of the joint 3b detected by the angle sensor 3c from the moving device 3, calculates the position of the tip of the electric holder 3a from the rotation angle of the joint 3b, and calculates the velocity Vtip of the tip of the electric holder 3a from the change in position over time. The velocity Vtip is a two-dimensional vector that represents the movement direction of the tip of the electric holder 3a in the YrZr plane perpendicular to the Xr axis, and represents the movement direction of the endoscope 2 in the holder coordinate system Σr.

[0027] The deviation calculation unit 54 calculates the deviation θ between the coordinate systems Σe and Σr using the movement vector Vobj and the velocity Vtip from the following equation (1). The deviation θ corresponds to the rotation angle θ of the endoscope 2 about the longitudinal axis A with respect to the electric holder 3a.

number

[0028] The coordinate system correction unit 55 corrects the holder coordinate system Σr based on the deviation θ to make the holder coordinate system Σr coincide with the endoscope coordinate system Σe. Specifically, the coordinate system correction unit 55 corrects the DH (Denavit-Hartenberg) parameters of the electric holder 3a based on the deviation θ so that the holder coordinate system Σr coincides with the endoscope coordinate system Σe. The control unit 56 controls the moving device 3 based on the corrected holder coordinate system Σr.

[0029] Next, the coordinate system correction method executed by the processor 5a will be described with reference to FIG. The user inputs a trigger to the user interface 4a at any timing when the user wishes to correct the holder coordinate system Σr. For example, the user manually rotates the endoscope 2 around the longitudinal axis A relative to the electric holder 3a in order to adjust the orientation of the subject S in the endoscopic image C displayed on the display device 6, and then inputs a trigger to the user interface 4a. Next, the user operates the operation device 4 to move the tip of the electric holder 3a and the endoscope 2.

[0030] In response to the user interface 4a receiving the trigger (YES in step S1), the processor 5a executes processes S2 to S12 for correcting the holder coordinate system Σr. Specifically, while the tip of the electric holder 3a and the endoscope 2 are moving, the processor 5a acquires an endoscopic image C at predetermined time intervals (step S2) and records the acquired endoscopic image C in the memory 5b or the storage unit 5c (steps S4, S6).

[0031] The endoscopic image C at the time the trigger is received is first acquired and stored as the first frame (YES in step S3, step S4), and the treatment tool area D in the endoscopic image C of the first frame is estimated by the treatment tool area estimation unit 51 (step S5). Thereafter, every time a new endoscopic image C is acquired (step S2), the newly acquired endoscopic image (i.e., the current endoscopic image) C is stored (step S6), a treatment tool area D in the current endoscopic image C is estimated by the treatment tool area estimation unit 51 (step S7), and a movement vector Vobj is detected by the subject movement detection unit 52 using the current endoscopic image C and the first frame endoscopic image C acquired in the past (step S8). Here, in step S8, the treatment tool area D is excluded from each of the two endoscopic images C of the current and first frame, and the movement vector Vobj is estimated based only on the other areas E of the two endoscopic images C.

[0032] If the magnitude of the movement vector Vobj is greater than or equal to the threshold value (YES in step S9), then the holder movement detection unit 53 detects the velocity Vtip of the tip of the electric holder 3a from the change in the position of the tip of the electric holder 3a over time from the time when the first frame of endoscopic image C is captured to the time when the current endoscopic image C is captured (step S10). Next, the deviation θ between the coordinate systems Σe and Σr is calculated by the deviation calculation unit 54 from the velocity Vtip and the movement vector Vobj (step S11). Next, the holder coordinate system Σr is corrected by the coordinate system correction unit 55 based on the deviation θ, so that the holder coordinate system Σr coincides with the endoscope coordinate system Σe (step S12).

[0033] On the other hand, if the magnitude of the motion vector Vobj is less than the threshold (NO in step S9), the process returns to step S2 without performing steps S10 to S12, and steps S2, S6, S7, and S8 are repeated until the magnitude of the motion vector Vobj becomes equal to or greater than the threshold. After step S12, the moving device 3 is controlled by the control unit 56 based on the corrected holder coordinate system Σr in accordance with the instruction input to the user interface 4a.

[0034] The endoscopic image C may include a moving subject (moving object) in addition to a stationary subject S. A moving subject affects the accuracy of the motion vector Vobj, and it is difficult to estimate the motion vector of the subject that accurately represents the moving direction of the endoscope 2 from an endoscopic image C that includes a moving subject. In particular, the treatment tool 7 moves in the endoscopic image C during surgery.

[0035] According to this embodiment, the region E of the endoscopic image C, excluding the treatment tool region D, is used to estimate the movement vector Vobj. As a result, even if a moving treatment tool 7 is present in the endoscopic image C, a highly accurate movement vector Vobj that accurately represents the movement direction of the endoscope 2 is estimated. Based on such a highly accurate movement vector Vobj, an accurate deviation θ between the coordinate systems Σe and Σr can be calculated, and the holder coordinate system Σr can be accurately corrected. Furthermore, based on the accurately corrected holder coordinate system Σr, the endoscope 2 can be moved by the movement device 3 in a direction that accurately corresponds to the direction input by the user to the operation device 4.

[0036] In addition, the deviation θ between the coordinate systems Σe and Σr is calculated only from two or more endoscopic images C and the information on the position of the tip of the electric holder 3a. In other words, there is no need to add a device such as a sensor to detect the deviation θ, and the deviation θ can be estimated sensorlessly. Therefore, there is no need to improve the electric holder 3a, and the electric holder 3a can be easily made thinner and more compact.

[0037] By adding a device such as a sensor that detects the rotation angle θ of the endoscope 2 to the electric holder 3a, the measurement of the deviation θ can be easily realized. However, in this case, the conventional electric holder cannot be used and the electric holder needs to be improved. In addition, the outer diameter, size and weight of the electric holder increase, and the electric holder 3a may get in the way of surgery.

[0038] In this embodiment, as shown in FIG. 4B, the treatment tool region estimation section 51 may estimate the moving direction of the subject S based on another region E' excluding the region obtained by adding a margin F to the treatment tool region D. The margin F is a region that extends along the contour of the treatment tool region D and surrounds the treatment tool region D, and is, for example, a band-shaped region having a predetermined width.

[0039] In the vicinity of the treatment tool 7, the movement of the subject S may be affected by the movement of the treatment tool 7. For example, the subject S may move partially in the vicinity of the treatment tool 7 by being pushed or pulled by the treatment tool 7. By excluding the region obtained by adding the margin F to the treatment tool region D, the estimation accuracy of the movement vector Vobj of the subject S can be improved, and the deviation θ between the coordinate systems Σe and Σr can be estimated more accurately.

[0040] Second embodiment Next, an endoscope system and a coordinate system correction method according to a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that the endoscope 21 is an oblique type. In this embodiment, differences from the first embodiment will be described, and configurations common to the first embodiment will be denoted by the same reference numerals and description thereof will be omitted.

[0041] The endoscope system according to this embodiment includes an endoscope 21, a moving device 3, an operating device 4, a control device 5, and a display device 6. As shown in FIG. 6, the endoscope 21 is an oblique-type rigid endoscope having a visual axis B inclined relative to a longitudinal axis A. The endoscope 21 has a long and rigid lens barrel portion 2a and an operation portion 2d connected to a base end of the lens barrel portion 2a. The operation portion 2d is rotatable about the longitudinal axis A relative to the lens barrel portion 2a, and the image sensor 2c is held inside the operation portion 2d and fixed to the operation portion 2d. The electric holder 3a holds the lens barrel portion 2a so that it can rotate about the longitudinal axis A. The image sensor 2c is a three-dimensional camera and can obtain a stereo image including three-dimensional position information of the subject S as an endoscopic image C.

[0042] When the lens barrel 2a rotates together with the operation unit 2d relative to the electric holder 3a, the visual axis B rotates about the longitudinal axis A, changing the direction of the visual axis B. When the operation unit 2d rotates relative to the lens barrel 2a and the electric holder 3a, the image pickup element 2c rotates about the longitudinal axis A, and the endoscopic image C displayed on the display device 6 rotates.

[0043] In this embodiment, the endoscope coordinate system Σe rotates relative to the holder coordinate system Σr due to the rotation of the barrel 2a relative to the electric holder 3a and the rotation of the operation unit 2d relative to the barrel 2a. Therefore, the deviation between the endoscope coordinate system Σe and the holder coordinate system Σr is calculated from the rotation angle θ of the barrel 2a around the longitudinal axis A and the rotation angle β of the operation unit 2d around the longitudinal axis A.

[0044] FIG. 7 illustrates a method for calculating the rotation angle θ of the lens barrel portion 2a and the rotation angle β of the operation portion 2d. An XbaseYbaseZbase reference coordinate system is set at the tip of the endoscope 21. The Xbase axis coincides with the longitudinal axis A. The coordinate system obtained by rotating the XbaseYbaseZbase reference coordinate system around the Xbase axis by θ is defined as the X1Y1Z1 coordinate system. Rotation around the Xbase axis corresponds to the rotation of the lens barrel section 2a. The coordinate system obtained by rotating the X1Y1Z1 coordinate system around the Y1 axis by the inclination angle α (for example, -30 deg) of the visual axis B is defined as the X2Y2Z2 coordinate system. The coordinate system obtained by rotating the X2Y2Z2 coordinate system around the X2 axis by β is defined as the X3Y3Z3 coordinate system. Rotation around the X2 axis corresponds to the rotation of the operation section 2d. The X3Y3Z3 coordinate system is the endoscope coordinate system Σe after rotation of the lens barrel section 2a and the operation section 2d.

[0045] Based on the above definition, the following formula is established.

number

[0046] where V = (vx, vy, vz). T is the three-dimensional velocity of the endoscope 21 in the holder coordinate system Σr, and is the velocity Vtip of the tip of the electric holder 3a detected by the holder movement detection unit 53. V'=(vx', vy', vz') T is the calculated velocity of the endoscope 21 in the endoscope coordinate system Σe, calculated based on the rotation angles θ, β and the velocity V. i i-1 is a homogeneous transformation matrix from the i-1 coordinate system to the i coordinate system. Also, Cθ=cosθ, Sθ=sinθ.

[0047] If the velocity of the endoscope 21 in the endoscope coordinate system Σe calculated based on the endoscopic image C is V'', then the rotation angle θ of the telescope tube portion 2a and the rotation angle β of the operating portion 2d can be estimated by finding the rotation angles θ and β that minimize the error between V' and V'', as shown in the following equation (2).

number

[0048] The velocity V" is the movement velocity Vobj of the subject S detected by the subject movement detection unit 52. In step S8, the subject movement detection unit 52 acquires three-dimensional position information of the subject S from each of the multiple endoscopic images C, estimates the position and posture of the endoscope 21 from the multiple endoscopic images C using a known image processing technique such as SfM (Structure from Motion) or Visual SLAM, and calculates the velocity V" = Vobj of the endoscope 21 in the endoscope coordinate system Σe from the amount of change in the estimated position and posture.

[0049] In step S11, the deviation calculation unit 54 comprehensively changes the rotation angles θ and β to calculate, as the deviation, the rotation angles θ and β that minimize the error between V' and V''. In step S12, the coordinate system correction unit 55 corrects the holder coordinate system Σr based on the rotation angles θ and β to make the holder coordinate system Σr coincide with the endoscope coordinate system Σe. Specifically, the coordinate system correction unit 55 corrects the DH (Denavit-Hartenberg) parameters of the electric holder 3a based on the rotation angles θ and β so that the holder coordinate system Σr coincides with the endoscope coordinate system Σe.

[0050] In this case, also in step S8 of this embodiment, the area E of the endoscopic image C excluding the treatment tool area D is used to estimate the movement vector Vobj. Therefore, even if a moving treatment tool 7 is present in the endoscopic image C, it is possible to estimate a highly accurate movement vector Vobj that accurately represents the movement direction of the oblique-viewing endoscope 21. Based on such a highly accurate movement vector Vobj, accurate deviations θ, β between the coordinate systems Σe, Σr can be calculated, and the holder coordinate system Σr can be accurately corrected. Other functions and effects of this embodiment are the same as those of the first embodiment, so description thereof will be omitted. In this embodiment, similarly to the first embodiment, the moving direction Vobj of the subject S may be estimated based on the other area E' excluding the area obtained by adding the margin F to the treatment tool area D.

[0051] Third embodiment Next, an endoscope system and a coordinate system correction method according to a third embodiment of the present invention will be described. This embodiment differs from the first embodiment in the method of detecting the moving direction of the subject S. In this embodiment, differences from the first embodiment will be described, and configurations common to the first embodiment will be denoted by the same reference numerals and description thereof will be omitted.

[0052] As in the first embodiment, the endoscope system 1 according to this embodiment includes an endoscope 2, a moving device 3, an operating device 4, a control device 5, and a display device 6. 8, in step S8, subject movement detection unit 52 estimates a movement vector u (see arrow) of each position in endoscopic image C, selects a movement vector u of a position in other area E excluding treatment tool area D, and estimates a movement vector Vobj of subject S from the selected movement vector u. For example, movement vector Vobj is an average of the selected movement vectors u.

[0053] 8 indicate positions where the motion vector u was not estimated due to the absence or scarcity of edge-like features (changes in gradation value). The positions where the motion vector was not estimated may also be excluded in estimating the motion vector Vobj of the subject S.

[0054] In an actual surgical scene, the subject S, such as an organ, in the other area E is not necessarily completely still, and may move partially in the other area E by being pushed or pulled by the treatment tool 7. Such a movement of the subject S causes an estimation error of the movement vector Vobj. According to this embodiment, by using the motion vectors u of a plurality of positions in the other region E, the estimation error of the motion vector Vobj of the subject S can be reduced.

[0055] Other functions and effects of this embodiment are the same as those of the first embodiment, so description thereof will be omitted. In this embodiment, similarly to the first embodiment, the movement direction Vobj of the subject S may be estimated based on another area E' excluding the area obtained by adding the margin F to the treatment tool area D. That is, the subject movement detection unit 52 may select a movement vector u of a position in the other area E', and estimate the movement vector Vobj of the subject S from the selected movement vector u.

[0056] (Fourth embodiment) Next, an endoscope system and a coordinate system correction method according to a fourth embodiment of the present invention will be described. This embodiment differs from the third embodiment in that only a motion vector u selected based on a direction from among the motion vectors u of positions in other region E is used to estimate a motion vector Vobj of subject S. In this embodiment, differences from the first and third embodiments will be described, and configurations common to the first and third embodiments will be denoted by the same reference numerals and description thereof will be omitted.

[0057] In step S8 of this embodiment, as shown in FIG. 9, the subject movement detection unit 52 selects a movement vector u for a position within another region E, and then further selects a movement vector u based on the direction di (i = 1, 2, 3, ...) of the movement vector u. Specifically, subject movement detection section 52 counts the number of movement vectors u ni (i=1, 2, 3, . . . ) for each direction di (i=1, 2, 3, . . . ).

[0058] Next, the subject movement detection unit 52 selects the most numerous motion vector u and a motion vector whose direction is close to that of the most numerous motion vector u. For example, a motion vector u whose direction is within ±δ° of the most numerous motion vector u is selected. δ is a value that is appropriately set. In the example of FIG. 9, the number n3 of motion vectors u in direction d3 is the most numerous, and the motion vector u in direction d3 and the motion vectors in directions d2 and d4 are selected. Next, the subject movement detection unit 52 estimates the movement vector Vobj of the subject S from only the selected movement vector u.

[0059] As described above, the subject S in the other area E may move partially regardless of the movement of the endoscope 2. According to this embodiment, the movement vectors u in the directions d1, d5, and d6 different from the movement direction d3 of the subject S caused by the movement of the endoscope 2 are excluded from the movement vectors u in the other regions E, and only the movement vectors u in the movement direction d3 of the subject S caused by the movement of the endoscope 2 and the directions d2 and d4 close to it are used to estimate the movement vector Vobj. This makes it possible to further reduce the estimation error of the movement vector Vobj of the subject S compared to the third embodiment.

[0060] Other functions and effects of this embodiment are the same as those of the first embodiment, so description thereof will be omitted. In this embodiment, similarly to the first embodiment, the moving direction Vobj of the subject S may be estimated based on the other area E' excluding the area obtained by adding the margin F to the treatment tool area D.

[0061] Fifth embodiment Next, an endoscope system and a coordinate system correction method according to a fifth embodiment of the present invention will be described. This embodiment differs from the third and fourth embodiments in that an enhancement process is applied to an endoscopic image C used to estimate a movement vector Vobj. In this embodiment, differences from the first, third and fourth embodiments will be described, and components common to the first, third and fourth embodiments will be denoted by the same reference numerals and description thereof will be omitted.

[0062] As described above, it is difficult to estimate the movement vector u at a position with no or few features. The subject movement detection unit 52 applies enhancement processing, such as edge enhancement processing or contrast enhancement processing, to the endoscopic image C in order to enhance features. For example, the subject movement detection unit 52 divides the endoscopic image C into a plurality of N×M regions, and applies enhancement processing to each region so that the histograms of the luminance values ​​of the plurality of regions become equal. If necessary, the subject movement detection unit 52 may apply pre-processing, such as grayscale conversion, to the color endoscopic image C before the enhancement processing.

[0063] Next, the subject movement detection unit 52 estimates a movement vector Vobj using two or more endoscopic images C that have been subjected to enhancement processing. Fig. 10 shows a movement vector u in an endoscopic image C that has been subjected to enhancement processing. As shown in Figs. 8 and 10, even in positions where a movement vector u was not estimated in an endoscopic image C that has not been subjected to enhancement processing, a movement vector u is estimated in an endoscopic image C that has been subjected to enhancement processing.

[0064] In this manner, according to this embodiment, by subjecting the endoscopic image C to processing for emphasizing features, the area in which the motion vector u is estimated is expanded, and the area contributing to the correct estimation of the motion vector Vobj of the subject S is increased. This makes it possible to further reduce the estimation error of the motion vector Vobj of the subject S, compared to the third and fourth embodiments.

[0065] Other functions and effects of this embodiment are the same as those of the first embodiment, so description thereof will be omitted. In this embodiment, similarly to the first embodiment, the moving direction Vobj of the subject S may be estimated based on the other area E' excluding the area obtained by adding the margin F to the treatment tool area D.

[0066] In the above first to fifth embodiments, the moving object is the treatment tool 7, but the moving object is not limited to the treatment tool 7 and may be any subject that moves within the endoscopic image C. For example, the moving object may be a surgical instrument such as a trocar or surgical tape, an organ such as a peristaltic intestine or a diaphragm that moves due to breathing, or smoke generated when tissue is cauterized by a treatment tool or the like.

[0067] In the above first to fifth embodiments, the treatment tool region estimation unit 51 estimates the region of a moving object such as the treatment tool 7 by image recognition using artificial intelligence, but instead, the region of a moving object may be estimated based on a movement vector u of each position in the endoscopic image C. For example, the treatment tool region estimation unit 51 may estimate a position where the magnitude of the movement vector u is equal to or greater than a threshold value as the region of a moving object.

[0068] This configuration also makes it possible to estimate the area of ​​the moving treatment tool 7. Furthermore, when another moving object is included in the endoscopic image C instead of or in addition to the treatment tool 7, the area of ​​any other moving object can also be estimated based on the movement vector u. Therefore, when another moving object is present in the endoscopic image C, a highly accurate movement vector Vobj that accurately represents the movement direction of the endoscope 2 can be estimated based on the other area excluding the area of ​​the other moving object.

[0069] In the above first to fifth embodiments, a deviation occurs between the coordinate systems Σe and Σr due to rotation of the endoscope 2, 21 about the longitudinal axis A relative to the moving device 3, but the present invention can also be applied to cases where a deviation occurs between the coordinate systems Σe and Σr due to any relative motion of the endoscope 2, 21 and the moving device 3. That is, a deviation in any direction between the coordinate systems Σe and Σr can be calculated from the movement direction in the holder coordinate system Σr of the endoscope 2 moved by the electric holder 3 and the movement direction of the subject S in the endoscopic image C. Therefore, based on the calculated deviation, at least one of the endoscope coordinate system Σe and the holder coordinate system Σr can be corrected so that the endoscope coordinate system Σe and the holder coordinate system Σr coincide with each other. [Explanation of symbols]

[0070] 1 Endoscope system 2,21 Endoscopy 2a Lens barrel part 2c Image sensor 2d operation section 3. Mobility Devices 3a Electric holder, robot arm 4 Control device 4a User Interface 5. Control device 5a Processor 7 Treatment tools (moving objects) A Longitudinal axis of the endoscope B. Visual axis of the endoscope C Endoscopic image D. Treatment tool area (moving body area) E,E' Other areas F Margin S Subject Σr Holder coordinate system (Moving device coordinate system) Σe Endoscope coordinate system (endoscope coordinate system)

Claims

1. An endoscope and a moving device for holding and moving the endoscope; one or more processors; the one or more processors Detecting a moving direction of the endoscope in a coordinate system of the moving device, the endoscope being moved by the moving device; Estimating a region of a moving object in an image captured by the endoscope; Detecting a direction of movement of a subject in the image based on other areas of the image excluding the area of ​​the moving object; calculating a deviation between a coordinate system of the endoscope and a coordinate system of the moving device based on a moving direction of the endoscope and a moving direction of the subject; The endoscope system corrects a coordinate system of the moving device based on the calculated deviation.

2. The moving device holds the endoscope rotatably about a longitudinal axis of the endoscope, The endoscopic system according to claim 1 , wherein the misalignment between the coordinate system of the endoscope and the coordinate system of the moving device includes a rotational misalignment about an axis corresponding to the longitudinal axis.

3. the one or more processors: Detecting a moving direction of the subject in the other area; The endoscope system according to claim 1 , wherein the deviation is calculated based on a moving direction of the endoscope and a moving direction of the subject in the other area.

4. the one or more processors: Detecting a motion vector of the object in the other region; The endoscope system according to claim 1 , wherein the deviation is calculated based on a moving direction of the endoscope and a moving vector of the subject in the other area.

5. a user interface for receiving a trigger input from a user; The endoscope system according to claim 1 , wherein the one or more processors execute a process for correcting a coordinate system of the mobile device in response to the user interface receiving the trigger.

6. the one or more processors detect a moving direction of the object based on two or more images captured at different times; The endoscope system according to claim 5 , wherein one of the two or more images is an image at the time when the trigger is received.

7. the one or more processors: acquiring images captured by the endoscope at predetermined time intervals; The endoscope system according to claim 5 , wherein each time the image is newly acquired, a moving direction of the subject is detected based on the newly acquired image and previously acquired images.

8. the one or more processors: Detecting a motion vector of the object; The endoscope system according to claim 1 , further comprising: a step of calculating the deviation and correcting the coordinate system of the moving device when a magnitude of the movement vector of the subject is equal to or greater than a predetermined threshold value.

9. the one or more processors: Detecting a plurality of motion vectors for a plurality of positions within the other region; The endoscope system according to claim 4 , further comprising: a motion vector of the subject calculated from the plurality of motion vectors.

10. the one or more processors: Detecting a plurality of movement directions of a plurality of positions within the other region; The endoscope system according to claim 3 , further comprising: a moving direction of the subject being calculated from the plurality of moving directions.

11. the one or more processors: Counting the number of the movement vectors for each direction; selecting a most numerous motion vector and a motion vector having a direction close to that of the most numerous motion vector from among the plurality of motion vectors; The endoscope system according to claim 9 , further comprising: a motion vector of the object being detected based on the selected motion vector.

12. the one or more processors: Counting the number of the movement directions for each direction; selecting a most common direction of movement and a direction of movement close to the most common direction of movement from among the plurality of directions of movement; The endoscope system according to claim 10 , further comprising: a moving direction of the subject being calculated from the selected moving direction.

13. the endoscope is an oblique-viewing endoscope having a long barrel portion and an operation portion connected to a base end of the barrel portion and holding an image sensor, the operation portion being rotatable around a longitudinal axis of the barrel portion with respect to the barrel portion, The endoscopic system according to claim 2, wherein the one or more processors calculate the deviation as a rotation angle of the endoscope relative to the moving device about the longitudinal axis and a rotation angle of the operating unit relative to the telescope barrel unit about the longitudinal axis based on the movement direction of the endoscope and the movement direction of the subject.

14. the endoscope is an oblique-viewing endoscope having a long barrel portion and an operation portion connected to a base end of the barrel portion and holding an image sensor, the operation portion being rotatable around a longitudinal axis of the barrel portion with respect to the barrel portion, the one or more processors: Obtaining three-dimensional position information of the subject within the image; The endoscope system according to claim 4 , wherein the three-dimensional movement vector is detected from the three-dimensional position information.

15. the endoscope is capable of acquiring a stereo image as the image, The endoscopic system of claim 14 , wherein the one or more processors acquire the three-dimensional position information from the stereo images.

16. the one or more processors: applying enhancement processing to the image to enhance features of the subject; The endoscope system according to claim 1 , further comprising: a moving direction of the subject being detected from the image on which the enhancement process has been performed.

17. The endoscope system according to claim 1 , wherein the one or more processors detect a moving direction of the subject based on the other area excluding an area including a margin added to the area of ​​the moving object.

18. the one or more processors: estimating a motion vector for each location in the image; The endoscope system according to claim 1 , wherein the region of the moving object is estimated based on a movement vector of each of the positions.

19. An endoscope and a moving device for holding and moving the endoscope; one or more processors; the one or more processors Detecting a movement direction of the endoscope in a coordinate system of the moving device, the endoscope being moved by the moving device; Estimating a motion vector for each position within an image captured by the endoscope; Detecting a direction of movement of an object in the image based on other areas of the image excluding areas where the magnitude of the movement vector is equal to or greater than a predetermined threshold; calculating a deviation between a coordinate system of the endoscope and a coordinate system of the moving device based on a moving direction of the endoscope and a moving direction of the subject; The endoscope system corrects a coordinate system of the moving device based on the calculated deviation.

20. Detecting a direction of movement of an endoscope moved by a moving device in a coordinate system of the moving device; Estimating a region of a moving object in an image captured by the endoscope; Detecting a direction of movement of a subject in the image based on other areas of the image excluding the area of ​​the moving object; calculating a deviation between a coordinate system of the endoscope and a coordinate system of the moving device based on a moving direction of the endoscope and a moving direction of the subject; A method for operating an endoscope system, comprising correcting a coordinate system of the moving device based on the calculated deviation.

Images