Endoscope system, endoscope movement control device, method, and program

The endoscope system uses image analysis to estimate rotation amounts and align coordinate systems, allowing intuitive and accurate movement despite axis misalignment, enhancing operational precision and device compactness.

JP2024169386APending Publication Date: 2024-12-05OLYMPUS CORPORATION(JP)
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2024083855
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-25
Filing Date
2024-05-23
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In endoscope systems where the longitudinal axis direction of the endoscope insertion section differs from the optical axis direction of the endoscope tip, it is difficult to intuitively and accurately move the endoscope in a desired direction while viewing the endoscopic image.

Method used

The endoscope system includes a cylindrical insertion portion with a rotatable image sensor and optical element, allowing for the estimation of rotation amounts based on image analysis before and after movement, enabling intuitive direction control through a processor that aligns the endoscope and holder coordinate systems.

Benefits of technology

Enables intuitive and accurate movement of the endoscope in the desired direction by aligning the coordinate systems, even when the longitudinal and optical axes are misaligned, without the need for additional sensors, thus maintaining precision and reducing the size of the moving device.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024169386000001_ABST
    Figure 2024169386000001_ABST
Patent Text Reader

Abstract

To provide an endoscope system capable of moving an endoscope in a desired direction intuitively from an endoscope image even when a longitudinal axis direction of an insertion part is different from an optical axis direction of the tip.SOLUTION: An endoscope system includes: an endoscope 2 equipped with an insertion part 2a, an image pick-up device 2e arranged at the base end so as to be rotatable around a longitudinal axis, and an optical element 2b whose optical axis is inclined in a direction different from the longitudinal axis direction; a moving device for moving the endoscope while holding it rotatably around the longitudinal axis; and a processor. The processor: transmitting a first signal for moving the endoscope in the longitudinal axis direction to the moving device; detecting a first moving direction M1 of a subject using images before and after the transmission; estimating a first rotation amount around the longitudinal axis of the optical element; transmitting a second signal for moving the endoscope in a direction orthogonal to the longitudinal axis direction to the moving device; detecting a second moving direction M2 of the subject using the images before and after the transmission; and estimating a second rotation amount around the longitudinal axis between the moving device and the endoscope on the basis of the second moving direction and the first rotation amount.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an endoscope system, an endoscope movement control device, a method, and a program. [Background technology]

[0002] Conventionally, there is known an endoscope system in which an endoscope is moved by a moving device such as an electric holder. In order to move the endoscope accurately, it is desirable that the coordinate system of the endoscope and the coordinate system of the moving device coincide with each other. For example, when an operator remotely controls an electric holder using a user interface, the operator inputs a desired movement direction of the endoscope to the user interface based on an endoscopic image. If the coordinate system of the endoscope and the coordinate system of the moving device match, the user can intuitively and accurately move the endoscope in the desired direction. On the other hand, if the coordinate system of the endoscope and the coordinate system of the moving device do not match, the actual movement direction of the endoscope differs from the movement direction input to the user interface, making it difficult for the user to intuitively and accurately move the endoscope in the desired direction.

[0003] In a robot system in which a camera system is attached to a movable arm, a method for correcting the transformation between the robot's coordinate system and the coordinate system of the camera system is known (see, for example, Patent Document 1). In Patent Document 1, an image of a target is acquired by the camera system, and the transformation is determined from the position of the movable arm when the image was acquired and the positions of feature points of the target in the image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,188,973 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the case of an endoscopic system in which the longitudinal axis direction of the insertion part of the endoscope differs from the optical axis direction of the endoscope tip, depending on the attachment angle of the endoscope to the movable arm and the optical axis direction of the endoscope tip, it can be difficult to intuitively move the endoscope in the desired direction while viewing the endoscopic image.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an endoscopic system, an endoscope rotation amount estimation method, and a program that enable intuitive movement of an endoscope in a desired direction while viewing an endoscopic image, even when the longitudinal axis direction of the insertion part of the endoscope differs from the optical axis direction of the tip of the endoscope. [Means for solving the problem]

[0007] One aspect of the present invention is an endoscope including a cylindrical insertion section extending in a longitudinal direction, an imaging element arranged at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element provided in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, a moving device that holds and moves the endoscope so as to be rotatable about the longitudinal axis, and at least one processor, wherein the processor transmits a first signal to the moving device for moving the endoscope in the longitudinal axis direction, and generates an image in the image using two images acquired by the imaging element before and after the transmission of the first signal. an endoscope system that detects a first movement direction of a subject in an image captured by the imaging element, estimates a first amount of rotation of the optical element around the longitudinal axis relative to the imaging element based on the first movement direction, transmits a second signal to the moving device to move the endoscope in a direction perpendicular to the longitudinal axis direction, detects a second movement direction of the subject in the image using two images acquired by the imaging element before and after transmitting the second signal, and estimates a second amount of rotation around the longitudinal axis between the moving device and the endoscope based on the second movement direction and the first amount of rotation.

[0008] Another aspect of the present invention is an endoscope movement control device that controls a moving device that moves an endoscope, the endoscope comprising a cylindrical insertion section extending in a longitudinal axis direction, an imaging element arranged at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element provided in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, the moving device holds the endoscope so as to be rotatable about the longitudinal axis and comprises at least one processor, the processor transmits a first signal to the moving device for moving the endoscope in the longitudinal axis direction, and acquires two images acquired by the imaging element before and after the transmission of the first signal. an endoscope movement control device that detects a first movement direction of the subject in the image using an image acquired by the imaging element, estimates a first rotation amount around the longitudinal axis of the optical element relative to the imaging element based on the first movement direction, transmits a second signal to the moving device to move the endoscope in a direction perpendicular to the longitudinal axis direction, detects a second movement direction of the subject in the image using two images acquired by the imaging element before and after transmitting the second signal, and estimates a second rotation amount around the longitudinal axis between the moving device and the endoscope based on the second movement direction and the first rotation amount.

[0009] Another aspect of the present invention is an endoscope movement control method for controlling a moving device that moves an endoscope, the endoscope comprising a cylindrical insertion section extending in a longitudinal direction, an image sensor arranged at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element provided in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, the moving device holds the endoscope so as to be rotatable about the longitudinal axis, moves the endoscope in the longitudinal axis direction using the moving device, and performs a first movement of a subject in the image using two images acquired by the image sensor before and after the movement in the longitudinal axis direction. a first movement direction of the subject relative to the imaging element based on the first movement direction; a second movement direction of the subject in the image using two images acquired by the imaging element before and after the movement in the direction perpendicular to the longitudinal axis direction; and a second movement amount of the subject in the image using the two images acquired by the imaging element before and after the movement in the direction perpendicular to the longitudinal axis direction.

[0010] Another aspect of the present invention is an endoscope movement control program for controlling a movement device that moves an endoscope, the endoscope comprising a cylindrical insertion section extending in a longitudinal direction, an image sensor arranged at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element provided in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, the movement device holds the endoscope so as to be rotatable about the longitudinal axis, moves the endoscope in the longitudinal axis direction using the movement device, and determines a first movement direction of a subject in the image using two images acquired by the image sensor before and after the movement in the longitudinal axis direction. and estimating a first amount of rotation of the optical element around the longitudinal axis relative to the imaging element based on the first direction of movement. The endoscope is moved in a direction perpendicular to the longitudinal axis direction by a moving device. The image sensor is used to detect a second direction of movement of the subject in the image using two images acquired by the imaging element before and after the movement in the direction perpendicular to the longitudinal axis direction. The endoscope movement control program is an endoscope movement control program that causes a computer to execute the following steps. Effect of the Invention

[0011] According to the present invention, even if the longitudinal axis direction of the insertion part of the endoscope differs from the optical axis direction of the tip of the endoscope, it is possible to intuitively move the endoscope in the desired direction while viewing the endoscopic image. [Brief description of the drawings]

[0012] [Figure 1] 1 is an overall configuration diagram of an endoscope system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the overall configuration of the endoscope system of FIG. 1. [Diagram 3] 1A and 1B are diagrams showing an example of a treatment tool region and other regions in an endoscopic image. [Figure 4] 2 is a partial longitudinal cross-sectional view showing an endoscope and a trocar of the endoscopic system of FIG. 1. [Figure 5A] 2 is a flowchart illustrating an endoscope movement control method performed by the endoscope system of FIG. 1. [Figure 5B] 5B is a flowchart continuing from FIG. 5A. [Figure 6] FIG. 11 is a diagram showing an example of a movement vector calculated from an endoscopic image. [Figure 7] FIG. 13 is a diagram showing an example of a temporary vanishing point set in an endoscopic image. [Figure 8] FIG. 2 is a diagram showing the relationship between the angle of view and the vanishing point of an endoscope. [Figure 9] 8A to 8C are diagrams for explaining the process of determining a vanishing point from the temporary vanishing point set in FIG. 7. [Figure 10] 11A and 11B are diagrams illustrating a process of calculating a rotation amount θ from a movement vector acquired from an endoscopic image before and after movement in a direction perpendicular to the longitudinal axis of the endoscope. [Figure 11] 1. FIG. 4 is a diagram showing a first modified example of the endoscope system of FIG. 1, illustrating an example of a treatment tool region, a margin, and other regions in an endoscope image. [Figure 12] 7 is a diagram illustrating a third modified example of the endoscope system in FIG. 1, and is a diagram for explaining a process of determining a vanishing point from the movement vector in FIG. 6. FIG. [Figure 13] FIG. 5B is a flowchart showing a fifth modified example of the endoscope system of FIG. 1, which follows FIG. 5A. [Figure 14] FIG. 13 is a flowchart showing a sixth modified example of the endoscope system of FIG. 1 and is an alternative to FIG. 5A. [Figure 15] 15 is a diagram illustrating the operation of the mobile device in response to a third signal in the flowchart of FIG. 14. FIG. [Figure 16] FIG. 5B is a flowchart showing a seventh modified example of the endoscope system of FIG. 1, which follows FIG. 5A. [Figure 17] FIG. 5B is a flowchart showing an eighth modified example of the endoscope system of FIG. 1, which follows FIG. 5A. [Figure 18] 1. FIG. 15 is a partial longitudinal sectional view of an endoscope and a trocar for explaining a ninth modified example of the endoscope system of FIG. [Figure 19]19 is a flowchart illustrating a registration process of a registration point in the endoscope system of FIG. 18. [Figure 20] 19 is a diagram for explaining distances between the tip of the endoscope in the endoscope system of FIG. 18 and multiple registration points and pivot points. [Figure 21] FIG. 19 is a flowchart illustrating an alternative to FIG. 5A for the endoscope system of FIG. 18. [Figure 22] 10A to 10C are diagrams showing the relationship between the angle of a bending joint, the angle of view, and the vanishing point of an endoscope in an endoscope system according to a second embodiment of the present invention. [Diagram 23] 23 is a diagram for explaining the amount of rotation about the longitudinal axis of the endoscope in the endoscope system of FIG. 22. FIG. [Figure 24] FIG. 23 is a flowchart illustrating an alternative to FIG. 5A for the endoscopic system of FIG. 22. [Diagram 25] 23A to 23C are diagrams for explaining a process for calculating the distance between the image center and the vanishing point and the amount of rotation in the endoscope system of FIG. 22. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] (First embodiment) An endoscope system 1, an endoscope movement control device 5, a method, a program, and a recording medium according to a first embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, the endoscopic system 1 of this embodiment includes an endoscope 2, a moving device 3 that holds and moves the endoscope 2, an operation device 4 operated by a user, a control device (endoscope movement control device) 5 that controls the moving device 3 based on an operation signal from the operation device 4, and a display device 6.

[0014] The endoscope 2 is a rigid endoscope having a long and rigid lens barrel portion (insertion portion) 2a, an optical system (optical element) 2b disposed within the lens barrel portion 2a, and a camera head 2c disposed at the base end of the lens barrel portion 2a. The endoscope 2 is an oblique-viewing endoscope 2 having a visual axis (optical axis) B tilted by a predetermined angle by the optical system 2b with respect to a longitudinal axis A extending along the radial center of the lens barrel portion 2a.

[0015] The lens barrel portion 2a is attached to the camera head 2c so as to be rotatable about the longitudinal axis A. An operating ring 2d is fixed to the lens barrel portion 2a. An operator can rotate the lens barrel portion 2a about the longitudinal axis A relative to the camera head 2c by holding the camera head 2c and rotating the operating ring 2d about the longitudinal axis A. Alternatively, the operator can rotate the lens barrel portion 2a about the longitudinal axis A relative to the camera head 2c by holding the operating ring 2d and rotating the camera head 2c about the longitudinal axis A.

[0016] An imaging element 2e that captures the light collected by the optical system 2b is fixed inside the camera head 2c. The imaging element 2e has an imaging surface that is arranged perpendicular to the longitudinal axis A and through which the longitudinal axis A passes. The imaging element 2e is an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0017] The endoscope 2 is inserted into the body together with one or more treatment tools 7, and an endoscopic image (image) C (see FIG. 3) including the one or more treatment tools 7 is acquired by the imaging element 2e and transmitted to the display device 6 via the control device 5. The display device 6 is any display such as a liquid crystal display or an organic EL display. The surgeon operates the treatment tool 7 while observing the endoscopic image C displayed on the display device 6.

[0018] The moving device 3 is a robot arm with five degrees of freedom having a plurality of, for example, three driving joints (joints) 3M and two passive joints 3P. The moving device 3 includes, for example, a base 3a installed on a horizontal plane, and a first link 3b that is rotationally driven with respect to the base 3a about a vertical first axis J1. The moving device 3 also includes a second link 3c that is rotationally driven with respect to the first link 3b about a horizontal second axis J2, and a third link 3d that is rotationally driven with respect to the second link 3c about a horizontal third axis J3. The moving device 3 also includes a fourth link 3e that is rotatably supported with respect to the third link 3d about a fourth axis J4 that is perpendicular to the third axis J3 and extends along a plane including the first axis J1, and a fifth link 3f that is rotatably supported with respect to the fourth link 3e about a fifth axis J5 that is perpendicular to the fourth axis J4.

[0019] The moving device 3 is provided with a holder 3g at the tip of the fifth link 3f, which holds the barrel portion 2a of the endoscope 2 rotatably around the longitudinal axis A. The holder 3g holds the barrel portion 2a so that the longitudinal axis A of the barrel portion 2a passes through the intersection of the fourth axis J4 and the fifth axis J5 and is perpendicular to the fifth axis J5. Although the moving device 3 has five degrees of freedom, since the holder 3g holds the barrel portion 2a rotatably around the longitudinal axis A, the degree of freedom becomes six overall, and the endoscope 2 can be placed in a desired posture and position.

[0020] The endoscope 2 moves due to the coordinated operation of the three driving joints 3M and two passive joints 3P of the moving device 3, and the position and attitude of the endoscope 2 are changed three-dimensionally. When the endoscope 2 is used, as shown in Fig. 4, the inside of the patient's abdominal cavity is observed by inserting the lens barrel portion 2a of the endoscope 2 into the through hole 8a of a trocar 8 that is attached to the abdominal wall W of the patient so as to penetrate the abdominal wall W. The trocar 8 is swingable about a pivot point X in the vicinity of the abdominal wall W as a fulcrum, and the inner diameter of the through hole 8a is formed to be slightly larger than the outer diameter of the lens barrel portion 2a. The gap between the through hole 8a and the lens barrel portion 2a is sealed by a sealing member made of an elastic material (not shown).

[0021] By driving the three driving joints 3M, the three-dimensional position of the tip of the third link 3d is uniquely determined. The movement direction of the endoscope 2 is restricted by the trocar 8. Therefore, when the tip position of the third link 3d is changed, the endoscope 2 swings the trocar 8 around the pivot point X or moves the lens barrel 2a in the direction of the longitudinal axis A within the through hole 8a of the trocar 8, passively rotating the two passive joints 3P to follow this. The rotation of the lens barrel 2a around the longitudinal axis A relative to the holder 3g is maintained in the state manually adjusted by the operator.

[0022] An angle sensor 3h for detecting the rotation angle of each of the three drive joints 3M of the moving device 3 is provided. The angle sensor 3h is, for example, an encoder, a potentiometer, or a Hall sensor provided in each drive joint 3M.

[0023] The operation device 4 has a user interface 4a including input devices such as keys, a joystick, buttons, and a touch panel. The operator can input instructions for moving the endoscope 2 to the operation device 4 by operating the user interface 4a. The operation device 4 transmits an operation signal based on the operation of the user interface 4a to the control device 5. The user interface 4a can also receive a trigger input from the operator. The trigger is for executing a process of estimating the rotation amounts β and θ of the endoscope 2, as will be described later.

[0024] As shown in FIG. 2, the control device 5 includes at least one processor 5a, a memory 5b, a storage unit 5c, an input interface 5d, and an output interface 5e. The control device 5 is connected to the surrounding endoscope 2, moving device 3, operating device 4 and display device 6 via an input interface 5d and an output interface 5e, and transmits and receives the endoscopic image C, information on the rotation angle of the driving joint 3M, and signals, etc. via the interfaces 5d, 5e.

[0025] The memory 5b is, for example, a semiconductor memory including a read-only memory (ROM) or a random access memory (RAM) area. The storage unit 5c is a non-transitory recording medium readable by a computer, and is, for example, a non-volatile recording medium including a hard disk or a semiconductor memory such as a flash memory. The processor 5a controls the movement device 3 based on an operation signal from the operation device 4, thereby moving the endoscope 2 according to an instruction input by the operator to the user interface 4a.

[0026] Here, the operator can rotate the endoscope 2 around the longitudinal axis A in two ways by operating the endoscope 2 attached to the holder 3g. First, by operating the operation ring 2d, the lens barrel portion 2a can be rotated around the longitudinal axis A relative to the camera head 2c, thereby changing the relative angle (first rotation amount) β between the camera head 2c and the lens barrel portion 2a. Second, by rotating the lens barrel portion 2a around the longitudinal axis A relative to the holder 3g, the angle (second rotation amount) θ representing the direction of the visual axis B of the lens barrel portion 2a relative to the holder 3g can be changed.

[0027] In order for an operator to attach the endoscope 2 to the moving device 3 and intuitively move the endoscope 2 up, down, left, and right while viewing the endoscopic image C displayed on the display device 6, it is necessary for the operator to understand the amounts of rotation β and θ between the moving device 3 and the endoscope 2. In the endoscope system 1 according to this embodiment, the processor 5a estimates the rotation amounts β and θ of the endoscope 2 by executing a program (endoscope movement control program) recorded in the storage unit 5c and read into the memory 5b.

[0028] Hereinafter, an endoscope movement control method using the endoscope system 1 according to this embodiment will be described with reference to the drawings. The process of estimating the rotation amounts β and θ of the endoscope 2 is started when the operator inputs a trigger to the user interface 4a.

[0029] As shown in FIG. 5A, the processor 5a first determines whether or not a trigger has been received (step S1). If a trigger has been received, the processor 5a β ,N θ Then, the processor 5a initializes N β It is determined whether N = 1 (step S3), β If N = 1, the first endoscopic image C acquired by the endoscope 2 at that time is stored in the memory 5b (step S4). The processor 5a also recognizes the treatment tool (moving object) 7 in the first endoscopic image C using a known method such as image recognition using artificial intelligence, and stores the area of ​​the treatment tool 7 as a treatment tool area D in the memory 5b (step S5). The processor 5a then repeats the loop count N β (step S6), and the process from step S3 is repeated.

[0030] In step S3, the processor 5a β If it is determined that ≠ 1, a first signal is transmitted to operate the movement device 3 so as to move the endoscope 2 in one direction along the longitudinal axis A (step S7). It is determined whether the movement amount of the endoscope 2 caused by the movement device 3 being operated by the first signal is equal to or greater than a predetermined threshold (step S8), and if it is equal to or greater than the threshold, the process ends.

[0031] When the processor 5a determines that the movement amount of the endoscope 2 is smaller than the threshold, the processor 5a stores the second endoscopic image C acquired by the endoscope 2 at that time in the memory 5b (step S9). The processor 5a also recognizes the treatment tool (moving object) 7 in the second endoscopic image C, and stores the area of ​​the treatment tool 7 as a treatment tool area D in the memory 5b (step S10).

[0032] Then, using the first endoscopic image C and the second endoscopic image C, a first moving direction M1 of the subject S in the endoscopic image C is detected (step S11). Specifically, the processor 5a uses a known method such as optical flow to detect a moving vector v of the subject S in the other area E excluding the treatment tool area D from the two endoscopic images C captured before and after the movement of the endoscope 2 by the moving device 3. obj Estimate the movement vector v obj are two-dimensional vectors indicating the movement directions of a plurality of different feature points in another region E in the endoscopic image C, as shown in FIG.

[0033] The processor 5a then calculates the estimated motion vector v obj It is determined whether the magnitude of the movement vector v is equal to or greater than a threshold (step S12). obj The magnitude of the estimated motion vector v obj The magnitude of the estimated displacement vector v obj This is done by using the average value of the magnitude of

[0034] Then, the processor 5a calculates the motion vector v obj If the magnitude of the motion vector v is smaller than the threshold, the process from step S7 is repeated. obj When the magnitude of the estimated motion vector v obj The amount of rotation β of the endoscope 2 is estimated from the above (step S13).

[0035] The amount of rotation β of the endoscope 2 is estimated as follows. First, the processor 5a calculates a plurality of motion vectors v estimated for a plurality of feature points on the endoscopic image C. obj Specifically, the processor 5a calculates the intersecting points (hereinafter also referred to as vanishing points) of the straight lines along the direction of the arrows. Specifically, as shown in FIG. 7, the processor 5a calculates the intersecting points Q1 to Q2 on a circumference of a circle with a radius L from the center O (hereinafter referred to as the image center) of the endoscopic image C. 12 are placed at equal intervals, and each temporary vanishing point Q1 to Q 12 The positions of the vanishing points Q1 to Q 12This is an example where the are arranged at 30° intervals.

[0036] The radius L is the distance from the image center O to the vanishing point, and can be calculated using formula (1).

number

[0037] The processor 5a sets the line connecting the position of the vanishing point Q1 when the rotation amount β=0° and the image center O as the reference line R, and calculates the temporary vanishing points Q1 to Q at the intersections of the line obtained by rotating the reference line R by, for example, 30° around the image center O and a circle of radius L. 12 As a result, for example, when a provisional vanishing point Q2 at a position rotated 30° with respect to the reference line R is selected as the vanishing point, the amount of rotation β of the endoscope 2 can be estimated as β=30°.

[0038] The processor 5a calculates each tentative vanishing point Q j For each movement vector v i and each movement vector v i The starting point p i and temporary vanishing point Q j Vector (Q j -p i 9 shows the inner product of the vector (Q2-p6) that connects the provisional vanishing point Q2 and the starting point p6 of the vector v6, and the provisional vanishing point Q 12 and the vector (Q 12 Then, the processor calculates all the motion vectors v i The sum of the inner products for the evaluation function U j The evaluation function U j The temporary vanishing point Q where j is selected as the vanishing point.

[0039]

number

[0040] Next, as shown in FIG. 5B, the processor 5a executes the loop count N θ But, N θ It is determined whether N=1 (step S14). θ If n=1, the processor 5a stores the third endoscopic image C acquired by the endoscope 2 at that time in the memory 5b (step S15). The processor 5a also stores the area of ​​the treatment tool (moving object) 7 in the third endoscopic image C as the treatment tool area D in the memory 5b in the same manner as in step S5 (step S16). The processor 5a then repeats the loop count N θ (step S17), and the process from step S14 is repeated.

[0041] In step S14, the processor 5a θ If it is determined that the rotation amount is not equal to 1, the processor 5a transmits a second signal to operate the moving device 3 so as to move the endoscope 2 in one direction perpendicular to the longitudinal axis A (step S18). The processor 5a determines whether the amount of movement of the endoscope 2 caused by operating the moving device 3 by the second signal is equal to or greater than a predetermined threshold (step S19), and if it is equal to or greater than the threshold, ends the rotation amount estimation process.

[0042] When the amount of movement of the endoscope 2 is smaller than the threshold value, the processor 5a uses the amount of rotation β estimated in step S13, and calculates an estimated movement vector v of the subject S in the endoscopic image C due to the transmission of the second signal when the amount of rotation θ is a predetermined value, for example, 0°. sys (Step S20). In FIG. 10, the movement vector v sysHere is an example: Then, the processor 5a stores in the memory 5b a fourth endoscopic image C acquired after the endoscope 2 is moved by the second signal (step S21). The processor 5a also recognizes the treatment tool (moving object) 7 in the fourth endoscopic image C, and stores the area of ​​the treatment tool 7 as a treatment tool area D in the memory 5b (step S22).

[0043] Then, using the third endoscopic image C and the fourth endoscopic image C, the processor 5a detects a second movement direction M2 of the subject S in the endoscopic image C (step S23). Specifically, the processor 5a uses a known method such as optical flow to detect a movement vector V of the subject S in the other area E excluding the treatment tool area D from the two endoscopic images C captured before and after the movement of the endoscope 2 by the moving device 3. real Estimate.

[0044] The processor 5a then calculates the estimated motion vector v real It is determined whether the magnitude of the movement vector v is equal to or greater than a threshold (step S24). real The magnitude of the estimated motion vector v real The magnitude of the estimated displacement vector v real This is done by using the average value of the magnitude of

[0045] Then, the processor 5a calculates the motion vector v real If the magnitude of the motion vector v is smaller than the threshold, the process from step S18 is repeated. real When the magnitude of the assumed movement vector v becomes equal to or larger than the threshold, the processor 5a sys and the estimated movement vector v real The angle between the endoscope 2 and the target object, which is 180° in the example shown in FIG. 10, is estimated as the amount of rotation θ of the endoscope 2 (step S25), and the process ends.

[0046] When the rotation amounts β and θ of the endoscope 2 are estimated, the processor 5a corrects the holder coordinate system Σr using the estimated rotation amounts β and θ so that the holder coordinate system Σr and the endoscope coordinate system Σe coincide with each other. The endoscope coordinate system Σe is a coordinate system fixed with respect to the imaging element 2e, and the holder coordinate system Σr is a coordinate system fixed with respect to the tip of the holder 3g.

[0047] In one example, the endoscope coordinate system Σe is an orthogonal coordinate system having mutually orthogonal Xe, Ye, and Ze axes, and the holder coordinate system Σr is an orthogonal coordinate system having mutually orthogonal Xr, Yr, and Zr axes. The Xe and Xr axes coincide with the longitudinal axis A, the Ye and Yr axes are parallel to the horizontal direction (left-right direction) of the endoscope image C, and the Ze and Zr axes are parallel to the vertical direction (up-down direction) of the endoscope image C. The holder coordinate system Σr and the endoscope coordinate system Σe coincide with each other means that the directions of the Xe and Xr axes coincide with each other, the directions of the Ye and Yr axes coincide with each other, and the directions of the Ze and Zr axes coincide with each other.

[0048] However, when the endoscope 2 is rotated by rotation amounts β, θ around the longitudinal axis A, a deviation occurs between the endoscope coordinate system Σe and the holder coordinate system Σr. When the endoscope coordinate system Σe does not match the holder coordinate system Σr in this manner, it is difficult for the operator observing the endoscopic image C displayed on the display device 6 to intuitively and accurately move the endoscope 2 in a desired direction by operating the user interface 4a.

[0049] The processor 5a corrects the holder coordinate system Σr based on the rotation amounts β and θ to make the holder coordinate system Σr coincide with the endoscope coordinate system Σe. Specifically, the processor 5a corrects the DH (Denavit-Hartenberg) parameters of the moving device 3 based on the rotation amounts β and θ so that the holder coordinate system Σr coincides with the endoscope coordinate system Σe. The processor 5a controls the moving device 3 based on the corrected holder coordinate system Σr.

[0050] In this way, according to this embodiment, the rotation amounts β and θ to be manually adjusted by the operator are estimated, and the holder coordinate system Σr is corrected using the estimated rotation amounts β and θ. Therefore, even if the endoscope 2 is an oblique viewer, the operator can intuitively and accurately move the endoscope 2 in a desired direction by operating the user interface 4a while observing the endoscopic image C.

[0051] Furthermore, the rotation amounts β and θ of the endoscope 2 can be calculated from two endoscopic images C acquired before and after operating the moving device 3. That is, there is no need to add devices such as sensors to detect the rotation amounts β and θ, and the rotation amounts β and θ can be estimated sensorlessly.

[0052] If a device such as a sensor for detecting the rotation amounts β and θ of the endoscope 2 is added to the moving device 3, the rotation amounts β and θ can be easily measured. However, in this case, the existing moving device 3 cannot be used and the moving device 3 needs to be improved. Furthermore, the size and weight of the moving device 3 will increase, and the moving device 3 may get in the way of surgery. Therefore, according to this embodiment, there is no need to improve the moving device 3 and the moving device 3 can be easily made smaller.

[0053] According to the present embodiment, the area E of the endoscopic image C, excluding the treatment tool area D, is moved by the movement vector v obj This allows a highly accurate motion vector v that accurately represents the moving direction of the endoscope 2 to be estimated even when a moving treatment tool 7 is present in the endoscopic image C. obj Such a highly accurate motion vector v obj Based on this, the rotation amounts β, θ of the endoscope 2 can be calculated, and the holder coordinate system Σr can be accurately corrected. Furthermore, based on the accurately corrected holder coordinate system Σr, the endoscope 2 can be moved by the movement device 3 in a direction accurately corresponding to the direction input by the operator to the operation device 4.

[0054] (First Modification) In this embodiment, as shown in FIG. 11, the treatment tool region estimation unit 51 may estimate the moving direction of the subject S based on another region E' excluding the region obtained by adding a margin F to the treatment tool region D. The margin F is a region that extends along the contour of the treatment tool region D and surrounds the treatment tool region D, and is, for example, a band-shaped region having a predetermined width.

[0055] In the vicinity of the treatment tool 7, the movement of the subject S may be affected by the movement of the treatment tool 7. For example, the subject S may move partially in the vicinity of the treatment tool 7 by being pushed or pulled by the treatment tool 7. By excluding the treatment tool area D plus the margin F, the movement vector v obj ,v real This improves the estimation accuracy, and makes it possible to estimate the rotation amounts β and θ of the endoscope 2 more accurately.

[0056] (Second Modification) In this embodiment, a plurality of temporary vanishing points Q corresponding to the rotation amount β are j and the evaluation function U j In this case, the tentative vanishing points are set every 30 degrees, but they may be set every smaller angle. This allows the amount of rotation β to be estimated with higher accuracy.

[0057] (Third Modification) In addition, multiple temporary vanishing points Q corresponding to the rotation amount β j Instead of setting, the movement vector v of multiple feature points in the endoscopic image C is obj Then, functions of a plurality of straight lines along the line may be obtained, and the intersection of these straight lines may be calculated as the vanishing point, as shown in Fig. 12. Then, the angle of the straight line connecting the calculated intersection point and the image center O with the reference line R may be calculated as the amount of rotation β. This allows the amount of rotation β to be estimated with even greater accuracy.

[0058] (Fourth Modification) As described above, a radial gap exists between the lens barrel 2a of the endoscope 2 and the through hole 8a of the trocar 8. When the moving device 3 has a passive joint 3P, even if the lens barrel 2a is moved in the direction of the longitudinal axis A, it may move in a direction intersecting the longitudinal axis A by the amount of the gap. Since the optical flow when the lens barrel 2a is moved in the direction of the longitudinal axis A is used to estimate the amount of rotation β, if the lens barrel 2a moves in a direction intersecting the longitudinal axis A, the estimation accuracy of the vanishing point and therefore the amount of rotation β decreases.

[0059] Therefore, it is preferable to move the lens barrel 2a in the direction of the longitudinal axis A by a distance sufficiently larger than the gap between the lens barrel 2a and the trocar 8 (for example, by 20 mm or more when the gap is 2 to 3 mm). obj As a result, the estimation accuracy of the rotation amount β can be improved.

[0060] (Fifth Modification) Moreover, in order to accurately position the lens barrel portion 2a of the endoscope 2 at a desired position and posture, a moving device 3 with six degrees of freedom is required. Although a robot arm having six degrees of freedom is generally used as the moving device 3, a moving device with three or more degrees of freedom can be used. In this embodiment, as described above, the holder 3g with six degrees of freedom holds the endoscope 2 so that it can be rotated manually around the longitudinal axis A. For this reason, for example, depending on the posture of the endoscope 2, when the lens barrel portion 2a of the endoscope 2 is moved in a direction perpendicular to the longitudinal axis A, the entire endoscope 2 may rotate around the longitudinal axis A. In this case, the endoscopic images C obtained before and after the movement rotate, making it difficult to correctly calculate the movement direction of the subject S.

[0061] Therefore, as shown in FIG. 13, the processor 5a performs the above-mentioned step S14. θIf it is determined that the target velocity of the tip of the mobile apparatus 3 is not equal to 1, the processor 5a sets a target velocity for operating the mobile apparatus 3 (step S26). Then, based on the set target velocity, the processor 5a assumes that the mobile apparatus 3 has six degrees of freedom, calculates the angular velocity of each of the joints 3M, 3P from inverse kinematics, and calculates a second signal for achieving the calculated angular velocity (step S27).

[0062] In this case, a second signal is also calculated for a joint that rotates the endoscope 2, which does not actually exist, about the longitudinal axis A. When the second signal calculated in this manner is transmitted to the moving device 3, if the moving device 3 has six degrees of freedom, the posture of the endoscope 2 is compensated by rotating the entire endoscope 2 about the longitudinal axis A. However, since the moving device 3 of this embodiment does not have a drive joint that rotates about the longitudinal axis A, no compensation is made.

[0063] Therefore, the processor 5a rotates the endoscopic image C stored in step S21 after the operation of the moving device 3 by a rotation angle corresponding to the second signal calculated for a joint about the longitudinal axis A that does not actually exist (step S28). As a result, the endoscopic image C acquired by the endoscope 2 can be the same as that acquired when the moving device 3 has six degrees of freedom, and at least one of the two endoscopic images C acquired before and after the transmission of the second signal is rotated by image processing so that the angles about the longitudinal axis A of the two endoscopic images C acquired before and after the transmission of the second signal match, thereby making it possible to correctly calculate the movement direction of the subject S before and after the movement.

[0064] (Sixth Modification) As described above, in this embodiment, the two joints at the tip of the moving device 3 are passive joints 3P and do not have motors or angle sensors. Therefore, when the three driving joints 3M are driven, the barrel portion 2a of the endoscope 2 swings the trocar 8, and is moved along the through hole 8a of the trocar 8 by the reaction force received from the trocar 8, and the passive joints 3P passively operate in response to this.

[0065] However, as described above, there is a gap between the inner surface of the through hole 8a of the trocar 8 and the outer surface of the lens barrel portion 2a of the endoscope 2. Therefore, as shown in Fig. 4, when the lens barrel portion 2a is not receiving a reaction force from the trocar 8, even if the endoscope 2 is moved in the direction of the longitudinal axis A by transmitting the first signal, the lens barrel portion 2a may move in a direction intersecting the direction of the longitudinal axis A after the movement starts. In this case, the correct movement vector v ovj Since the rotation amount β of the endoscope 2 cannot be obtained, the estimation accuracy of the rotation amount β of the endoscope 2 decreases.

[0066] Therefore, as shown in Fig. 14, before acquiring the first endoscopic image C, the processor 5a may transmit a third signal to the moving device 3 (step S30) to move the endoscope 2 to a position where the lens barrel portion 2a receives a reaction force of a predetermined threshold value or more from the trocar 8. For example, by transmitting the third signal, as shown in Fig. 15, the inclination angle of the lens barrel portion 2a may be changed to elastically deform a sealing member (not shown) between the lens barrel portion 2a and the through-hole 8a and / or the abdominal wall W, thereby generating a reaction force of a predetermined threshold value or more. By performing the process from step S3 thereafter, the amount of rotation β can be accurately estimated.

[0067] (Seventh Modification) Furthermore, when the moving device 3 has a passive joint 3P, problems may arise when estimating the amount of rotation θ of the endoscope 2. That is, when there is a gap between the lens barrel 2a and the through hole 8a of the trocar 8, even if the endoscope 2 is attempted to be moved straight in a direction perpendicular to the longitudinal axis A, the passive joint 3P may move by itself by the amount of the gap, making it impossible to move the endoscope 2 straight.

[0068] Therefore, as shown in FIG. 16, in the above-mentioned step S14, N θ If it is determined that the value is not 1, the processor 5a calculates a second signal for realizing the movement direction of the endoscope 2 in which the passive joint 3P moves to move the endoscope 2 straight (step S31). Then, the processor 5a may transmit the second signal thus calculated to the movement device 3. Since the endoscope 2 is moved in the direction in which the passive joint 3P moves, the endoscope 2 can be moved straight in a direction perpendicular to the longitudinal axis A, and the estimation accuracy of the rotation amount θ can be improved.

[0069] (Eighth Modification) In this embodiment, in step S20, the assumed movement vector v sys In step S25, the movement vector v sys The estimated movement vector v real The angle between the angle θ and the angle θ is estimated as the amount of rotation θ of the endoscope 2. Alternatively, as shown in FIG. 17, the processor 5a performs simulations for a plurality of amounts of rotation θ to calculate the movement vector of the subject S on the endoscopic image C, and estimates the actual movement vector v real The amount of rotation θ used in the simulation in which the movement direction coincides with the direction of rotation θ may be used as the estimated result.

[0070] That is, first, the processor 5a stores the angles of the drive joints 3M of the moving device 3 at the time when the first endoscopic image C and the treatment tool area D are stored in steps S15 and S16 (step S32). Then, the processor 5a calculates a sufficiently large actual movement vector v real After the rotation amount θ is calculated, the processor 5a again stores the angles of the drive joints 3M of the moving device 3 (step S33). After that, the processor 5a calculates the movement vector v simu (Step S34). Then, the processor 5a calculates the actual movement vector v real The movement vector v whose movement direction is the same as simu The simulation in which the rotation amount θ was calculated is selected, and the rotation amount θ used in that simulation is adopted as an estimated value (step S35).

[0071] In the simulation, the angles of the joints 3M stored in steps S32 and S33, the amount of rotation β estimated in step S13, the lengths of the links 3b, 3c, and 3d, and the distance from the tip of the endoscope 2 to the subject S are used as fixed values. Then, the processor 5a calculates the movement vector v of the subject S using the amount of rotation θ as a parameter. simu We perform multiple simulations to calculate

[0072] According to this embodiment, the simulation is performed using the actual angles of each driving joint 3M of the moving device before and after the movement, so the movement vector v is calculated by taking into account the fact that the moving device has five degrees of freedom and the change in the attitude of the endoscope 2 due to the presence of the passive joint 3P. simu Therefore, the movement vector v in the first embodiment, which is calculated on the assumption that the tip of the endoscope 2 moves straight in a direction perpendicular to the longitudinal axis A, can be calculated as follows: sys Compared with the movement vector v simu can be calculated with high accuracy, and the estimation accuracy of the rotation amount θ can be improved.

[0073] Moreover, although the sixth to eighth modified examples are implemented individually in the above description, at least two of them may be implemented in combination instead.

[0074] (Ninth Variation) Furthermore, when inserting the endoscope 2 via the trocar 8 as described above, if the endoscope 2 is moved too far forward along the longitudinal axis A to estimate the amount of rotation β, the tip of the endoscope 2 may come into contact with the subject S in the abdominal cavity. On the other hand, if the endoscope 2 is moved too far back along the longitudinal axis A, the tip of the endoscope 2 may enter the trocar 8, and sufficient optical flow may not be obtained.

[0075] 18, the operator operates the movement device 3 while checking the endoscopic image C to move the endoscope 2 to a position and posture where the tip does not come into contact with the subject S inside the body cavity, and registers the three-dimensional position of the tip of the endoscope 2 at that time as a registration point T. Then, the process of estimating the rotation amount β may be performed by moving the endoscope 2 in a direction retracting along the longitudinal axis A from a state in which the tip of the endoscope 2 is positioned at the registration point T.

[0076] Specifically, as shown in Fig. 19, the processor 5a registers the three-dimensional position of the tip of the endoscope 2 as a registration point T (step S36). There may be one or more registration points T. As shown in Fig. 20, when registering multiple registration points T, the processor 5a determines whether or not to end the registration (step S37) and repeats the process of step S36 an arbitrary number of times.

[0077] Then, when the estimation process of the rotation amounts β and θ is started with the tip of the endoscope 2 placed at an arbitrary position, the processor 5a calculates the distance L1 between the tip of the endoscope 2 and the pivot point X (step S38), as shown in Fig. 21. The processor 5a then determines whether the distance L1 is equal to or smaller than a threshold value (step S39), and if the distance L1 is greater than the threshold value, executes the process from step S2. In this case, the movement direction of the moving device 3 according to the first signal transmitted in step S7 may be either the direction of advancing or retracting the endoscope 2 along the longitudinal axis A.

[0078] If it is determined in step S39 that the distance L1 is equal to or less than the threshold, the processor 5a calculates the distance L2 between the tip of the endoscope 2 and the registered point T for all registered points T (step S40). Then, the processor 5a moves the tip of the endoscope 2 to the registered point T for which the calculated distance L2 is the smallest (step S41), and then performs the process from step S2. In this case, the operation direction of the moving device 3 is limited to the direction in which the endoscope 2 is moved backward along the longitudinal axis A by the first signal transmitted in step S7.

[0079] In this way, according to this embodiment, the rotation amount β can be efficiently estimated by preventing the inconvenience of entering the trocar 8 during the estimation process of the rotation amount β. In particular, by simply registering the registration point T, the estimation process can be performed regardless of the tip position of the endoscope 2 at the start of estimation. This has the advantage that it is possible to eliminate the need for the operator to manually move the endoscope 2 to a position where the distance L1 is greater than the threshold value each time.

[0080] Furthermore, when the tip of the endoscope 2 is moved to the registration point T by operating the movement device 3, it is preferable that the processor 5a moves the tip of the endoscope 2 as follows. First, the endoscope 2 is moved backward along the longitudinal axis A until the distance L1 between the tip of the endoscope 2 and the pivot point X becomes equal to or less than a certain distance. Next, the attitude of the endoscope 2 is adjusted so that the longitudinal axis A of the endoscope 2 is parallel to a line connecting the registration point T to be moved to and the pivot point X. Finally, the endoscope 2 is moved along the longitudinal axis A until the distance between the tip of the endoscope 2 and the registration point T is equal to or less than a predetermined threshold value. As a result, even if an obstacle such as tissue or a treatment tool is present between the tip of the endoscope 2 at the start of the estimation process of the rotation amount β and the registration point T to be moved to, the tip of the endoscope 2 can be moved to the registration point T without coming into contact with the obstacle.

[0081] In addition, in this embodiment, the moving device 3 having three driving joints 3M and two passive joints 3P is exemplified. Alternatively, a moving device 3 having five or more driving joints 3M may be adopted. This allows the position and posture of the endoscope 2 to be controlled with higher accuracy. However, the driving joints 3M require a motor and an angle sensor, so the moving device 3 becomes large. Therefore, according to this embodiment, by providing the minimum number of driving joints 3M that can achieve the position and posture of the endoscope 2, the moving device 3 can be made small and the moving device 3 can be prevented from interfering with surgery.

[0082] In this embodiment, the imaging element 2e in the camera head 2c is arranged so as to be perpendicular to the longitudinal axis A of the lens barrel portion 2a and so as to pass through the center of the imaging surface. Alternatively, when the optical axis B is tilted by an optical element 2b such as a mirror or a prism in the camera head 2c, the imaging element 2e may be arranged so as to be perpendicular to the optical axis B tilted with respect to the longitudinal axis A and so as to pass through the center of the imaging surface.

[0083] Second embodiment Next, an endoscope system 1, an endoscope movement control device 5, a method, a program, and a recording medium according to a second embodiment of the present invention will be described. In this embodiment, differences from the first embodiment will be described, and the same reference numerals will be used to designate the same components as in the first embodiment, and the description thereof will be omitted.

[0084] The endoscope system 1 according to this embodiment differs from the first embodiment in that the endoscope 2 has a bending joint 2f with two degrees of freedom at the tip of the lens barrel portion 2a, instead of the optical element 2b that is provided on the lens barrel portion 2a and tilts the visual axis B, as shown in Figures 22 and 23. Since the bending joint 2f has two degrees of freedom, there is no operation ring 2d for rotating the tilt direction of the visual axis B caused by the optical element 2b around the longitudinal axis A.

[0085] In the description of this embodiment, the tilt angle of the visual axis B relative to the longitudinal axis A of the lens barrel portion 2a due to bending of the bending joint 2f is defined as the rotation amount γ as shown in Fig. 22, and the angle indicating the bending direction of the bending joint 2f centered on the longitudinal axis A is defined as the rotation amount δ as shown in Fig. 23. These rotation amounts γ and δ define two degrees of freedom of the bending joint 2f of the endoscope 2. Also, the rotation amount θ is the mounting angle of the lens barrel portion 2a of the endoscope 2 to the holder 3g, as in the first embodiment, and can be manually changed as desired by the operator without using the movement device 3.

[0086] In the endoscope system 1 according to this embodiment, the processor 5a estimates the rotation amounts γ, δ, and θ of the endoscope 2 by executing a program (endoscope movement control program) recorded in the storage unit 5c and read into the memory 5b.

[0087] Hereinafter, an endoscope movement control method using the endoscope system 1 according to this embodiment will be described with reference to the drawings. As shown in FIG. 24, in the endoscope movement control method according to the present embodiment, the loop count N β Instead of loop count N γδ The process up to step S12 is performed in the same manner as in the first embodiment, except that the process differs in that the first embodiment uses the same formula as in the first embodiment.

[0088] This embodiment differs from the first embodiment in that the rotation amounts γ and δ are estimated (step S45). The amount of rotation γ can be calculated by equation (3) using θ=0° in the positional relationship shown in FIG. 22 and the distance L between the vanishing point and the image center O.

number

[0089] Here, the distance L is calculated as follows. First, as shown in FIG. 25, a motion vector v of two or more feature points on an endoscopic image C calculated in step S11 is obj Using the above formula, we find the equations of the straight lines that indicate the direction of each vector, and from these equations of the straight lines, we find the intersection of the straight lines as the vanishing point. This allows us to find the distance L from the image center O to the vanishing point.

[0090] The amount of rotation γ can be estimated by substituting the calculated distance L into equation (3). The amount of rotation δ can be estimated as the angle of the straight line connecting the calculated vanishing point and the image center O with respect to the reference line R. The amount of rotation θ can be estimated in the same manner as in the first embodiment.

[0091] As described above, according to this embodiment, the rotation amounts γ and δ of the two degrees of freedom of the bending joint 2f of the endoscope 2 that is manually adjusted by the operator and the rotation amount θ that is the mounting angle of the endoscope 2 that is manually adjusted are estimated, and the holder coordinate system Σr is corrected using the estimated rotation amounts γ, δ, and θ. Therefore, even in the case of an endoscope 2 that has a bending joint 2f, the operator can intuitively and accurately move the endoscope 2 in a desired direction by operating the user interface 4a while observing the endoscopic image C. [Explanation of symbols]

[0092] 1 Endoscope system 2 Endoscope 2a Lens tube part (insertion part) 2b Optical system (optical elements) 2c Camera Head 2d operating ring 2e Image sensor 3. Mobility Devices 3M Drive joint (joint) 3P passive joint 5 Control device (endoscope movement control device) 5a processor 8 Trocca S Subject A Longitudinal axis B optical axis M1 First movement direction M2 Second movement direction β First rotation amount θ Second rotation amount

Claims

1. an endoscope including a cylindrical insertion section extending in a longitudinal direction, an imaging element disposed at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element disposed in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction; a moving device that holds and moves the endoscope rotatably around the longitudinal axis; At least one processor; The processor: Transmitting a first signal to the moving device to move the endoscope in the longitudinal direction; detecting a first moving direction of a subject in an image by using two images acquired by the imaging element before and after the transmission of the first signal; estimating a first amount of rotation of the optical element about the longitudinal axis relative to the imaging element based on the first direction of movement; Transmitting a second signal to the moving device to move the endoscope in a direction perpendicular to the longitudinal axis direction; detecting a second movement direction of the subject in the image using two images acquired by the imaging element before and after the transmission of the second signal; an endoscope system that estimates a second amount of rotation about the longitudinal axis between the moving device and the endoscope based on the second direction of movement and the first amount of rotation.

2. The processor, detecting a moving direction of a plurality of feature points of the subject in the image as the first moving direction, and calculating positions of intersections of straight lines along the first moving direction of each of the plurality of feature points; The endoscope system according to claim 1 , wherein the first rotation amount is estimated based on a position of the intersection point.

3. The processor, The endoscope system according to claim 1 , wherein the second rotation amount is estimated in consideration of a difference between a direction in which the movable device has moved by transmitting the second signal and the second moving direction.

4. The processor, 2. The endoscope system according to claim 1, wherein a movement direction of the subject in the image due to transmission of the second signal is calculated by simulating a plurality of patterns in which the second rotation amount is a variable parameter, and a value of the variable parameter in a simulation result that coincides with the second movement direction is estimated as the second rotation amount.

5. The imaging element has an imaging surface disposed perpendicular to the longitudinal axis direction, The endoscopic system of claim 1 , wherein the longitudinal axis passes through an imaging center of the imaging surface.

6. The movement device includes a plurality of joints; The endoscopic system according to claim 5, wherein the tip of the insertion section can be moved to a predetermined position in three-dimensional space by driving at least one of the plurality of joints using a signal from the processor, while the attitude of the insertion section can be controlled except for rotation around the longitudinal axis.

7. The processor, The endoscopic system according to claim 6, wherein the second direction of movement is detected by rotating at least one of the two images acquired before and after the transmission of the second signal by image processing so that the angles around the longitudinal axis of the two images acquired before and after the transmission of the second signal are consistent.

8. the plurality of joints includes three joints that can be driven by the signal and two passive joints; The insertion portion is inserted into a trocar, The processor, 7. The endoscope system according to claim 6, further comprising: a third signal that is transmitted to the moving device to change the tilt angle in the longitudinal axis direction until the insertion section receives a reaction force from the trocar that is equal to or greater than a predetermined threshold, and then the first moving direction is detected.

9. the endoscope includes an operating ring fixed to the insertion portion, The endoscope system according to claim 6 , wherein the first rotation amount is a change amount of a rotation angle of the operation ring about the longitudinal axis with respect to the imaging element.

10. The endoscope system according to claim 1 , wherein the processor calculates the first movement direction and the second movement direction by optical flow.

11. An endoscope movement control device that controls a movement device that moves an endoscope, the endoscope comprises a cylindrical insertion section extending in a longitudinal direction, an imaging element disposed at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element disposed in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, the moving device holds the endoscope rotatably about the longitudinal axis; At least one processor; The processor: Transmitting a first signal to the moving device to move the endoscope in the longitudinal direction; detecting a first moving direction of a subject in an image by using two images acquired by the imaging element before and after the transmission of the first signal; estimating a first amount of rotation of the optical element about the longitudinal axis relative to the imaging element based on the first direction of movement; Transmitting a second signal to the moving device to move the endoscope in a direction perpendicular to the longitudinal axis direction; detecting a second movement direction of the subject in the image using two images acquired by the imaging element before and after the transmission of the second signal; an endoscope movement control device that estimates a second amount of rotation about the longitudinal axis between the movement device and the endoscope based on the second direction of movement and the first amount of rotation.

12. An endoscope movement control method for controlling a moving device that moves an endoscope, comprising: the endoscope comprises a cylindrical insertion section extending in a longitudinal direction, an imaging element disposed at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element disposed in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, the moving device holds the endoscope rotatably about the longitudinal axis; moving the endoscope in the longitudinal direction by the moving device; Detecting a first movement direction of the subject in the image using two images acquired by the image sensor before and after the movement in the longitudinal axis direction; estimating a first amount of rotation of the optical element about the longitudinal axis relative to the imaging element based on the first direction of movement; moving the endoscope in a direction perpendicular to the longitudinal axis direction by a moving device; Detecting a second movement direction of the subject in the image using two images acquired by the image sensor before and after the movement in a direction perpendicular to the longitudinal axis direction; and estimating a second amount of rotation about the longitudinal axis between the moving device and the endoscope based on the second direction of movement and the first amount of rotation.

13. An endoscope movement control program for controlling a moving device that moves an endoscope, the endoscope comprises a cylindrical insertion section extending in a longitudinal direction, an imaging element disposed at a base end of the insertion section so as to be rotatable about the longitudinal axis, and an optical element disposed in the insertion section and tilting an optical axis in a direction different from the longitudinal axis direction, the moving device holds the endoscope rotatably about the longitudinal axis; moving the endoscope in the longitudinal direction by the moving device; Detecting a first movement direction of the subject in the image using two images acquired by the image sensor before and after the movement in the longitudinal axis direction; estimating a first amount of rotation of the optical element about the longitudinal axis relative to the imaging element based on the first direction of movement; moving the endoscope in a direction perpendicular to the longitudinal axis direction by a moving device; Detecting a second movement direction of the subject in the image using two images acquired by the image sensor before and after the movement in a direction perpendicular to the longitudinal axis direction; and estimating a second amount of rotation about the longitudinal axis between the moving device and the endoscope based on the second direction of movement and the first amount of rotation.

Citation Information

Patent Citations

  • Calibration and transformation of a camera system's coordinate system

    US9188973B2