Endoscopic system, control device, control method and recording medium

Abstract

The endoscope system (10) includes an endoscope (2), a moving device (3) for moving the endoscope (2), a storage unit, and a control device (4) with a processor. The storage unit stores first position information and first rotation angle information of a first region within the subject, and second position information and second rotation angle information of a second region within the subject. The first rotation angle information defines the rotation angle of the endoscopic image of the first region, and the second rotation angle information defines the rotation angle of the endoscopic image of the second region. The processor calculates the third rotation angle information of the third region based on the first position information, the second position information, the first rotation angle information, the second rotation angle information, and the third position information of the third region within the subject. When the current imaging area is included in the third region, the processor rotates the endoscopic image based on the third rotation angle information and outputs the rotated endoscopic image to the display device.

Description

Technical Field

[0001] This invention relates to endoscope systems, control devices, control methods, and recording media.

[0002] This application claims priority based on U.S. Provisional Application No. 63 / 076,408, filed provisionally in the United States on September 10, 2020, the contents of which are incorporated herein by reference. Background Technology

[0003] Previously, there were known endoscope systems that moved an endoscope held in a retainer by controlling an electrically operated retainer (for example, see Patent Document 1).

[0004] The endoscope system in Patent Document 1 stores the time-series changes in the rotation angles of each joint of the retainer during the operator's movement of the endoscope in manual mode. In automatic recovery mode, the time-series changes in the rotation angles of each joint are inversely reproduced. As a result, the endoscope moves in the opposite direction to the movement trajectory in manual mode, automatically returning to its initial position and orientation.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent No. 6161687 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] When a surgeon performs surgery on a affected area while observing an endoscopic image displayed on a display device, the vertical orientation of the endoscopic image on the display device is important. The vertical orientation of an endoscopic image represents the angle of rotation of the subject within the endoscopic image; it is also referred to as the geocentric direction in other descriptions. The arrangement of organs and the appearance of tissues differ depending on the vertical orientation of the endoscopic image. Therefore, in order for the surgeon to accurately identify organs and tissues within the endoscopic image on the display device, it is preferable to display an endoscopic image with an appropriate vertical orientation on the display device.

[0010] However, as the endoscope moves within the body, the vertical orientation of the endoscopic image changes. Furthermore, the preferred vertical orientation of the endoscopic image varies for the surgeon depending on the observation position and the nature of the surgery. The endoscopic system in Patent Document 1 only reproduces the movement trajectory in manual mode and lacks the function of adjusting the vertical orientation of the endoscopic image based on the endoscope's position or the observation position. Therefore, in order to adjust the vertical orientation of the endoscopic image, the surgeon needs to temporarily release the instrument being operated on, interrupting the procedure, and manually manipulate the endoscope to adjust its posture.

[0011] The present invention was made in view of the above circumstances, and its object is to provide an endoscope system, control device, control method and recording medium capable of automatically adjusting the vertical direction of an endoscopic image.

[0012] Methods for solving problems

[0013] One aspect of the present invention is an endoscope system comprising: an endoscope inserted into a subject body (within a patient's body cavity) and acquiring endoscopic images within the subject body; a moving device for holding and moving the endoscope; a storage unit; and a control unit having at least one processor, the storage unit storing first position information and first rotation angle information of a first region within the subject body, and second position information and second rotation angle information of a second region within the subject body different from the first region, wherein the first rotation angle information is information defining the rotation angle of the endoscopic image of the first region, and the second rotation angle information is information defining the rotation angle of the endoscopic image of the first region. The processor determines the rotation angle information of the endoscopic image in the second region, and performs the following processing: calculating the third rotation angle information of the third region based on the first position information, the first rotation angle information, the second position information, the second rotation angle information, and the third position information of the third region in the subject body, wherein the third region is a region different from the first region and the second region, and when the current imaging area captured by the endoscope is included in the third region, rotating the endoscopic image according to the third rotation angle information, and outputting the rotated endoscopic image to the display device.

[0014] Another aspect of the present invention is a control device for controlling endoscopic images acquired by an endoscope and displayed on a display device. The control device includes a storage unit and at least one processor. The storage unit stores first position information and first rotation angle information of a first region within a subject, and second position information and second rotation angle information of a second region within the subject that is different from the first region. The first rotation angle information defines the rotation angle of the endoscopic image of the first region, and the second rotation angle information defines the rotation angle of the endoscopic image of the second region. The at least one processor performs the following processing: calculating a third rotation angle information of a third region based on the first position information, the first rotation angle information, the second position information, the second rotation angle information, and the third position information of a third region within the subject, wherein the third region is different from the first and second regions; when the current imaging area captured by the endoscope is included in the third region, rotating the endoscopic image based on the third rotation angle information, and outputting the rotated endoscopic image to the display device.

[0015] Another aspect of the present invention is a control method that uses first position information and first rotation angle information of a first region within a subject, and second position information and second rotation angle information of a second region within the subject that is different from the first region, to control an endoscopic image acquired by an endoscope and displayed on a display device. The first rotation angle information defines the rotation angle of the endoscopic image of the first region, and the second rotation angle information defines the rotation angle of the endoscopic image of the second region. The control method includes the following steps: calculating a third rotation angle of a third region based on the first position information, the first rotation angle information, the second position information, the second rotation angle information, and a third position information of a third region within the subject, wherein the third region is a region different from the first and second regions; rotating the endoscopic image based on the third rotation angle information when the current imaging area captured by the endoscope is contained within the third region; and outputting the rotated endoscopic image to the display device.

[0016] Another aspect of the invention is a computer-readable, non-transitory recording medium containing a control program for causing a computer to perform the control method described above.

[0017] Invention Effects

[0018] According to the present invention, it is effective in providing the operator with appropriate vertical endoscopic images based on the observation position within the subject. Attached Figure Description

[0019] Figure 1A This is an external diagram showing the overall structure of the endoscope system.

[0020] Figure 1B This diagram illustrates the movement of an endoscope inserted into the abdominal cavity.

[0021] Figure 1C This is a diagram showing the front end of the robotic arm and the endoscope.

[0022] Figure 2 It is a block diagram representing the overall structure of the endoscope system.

[0023] Figure 3A This is a timing diagram illustrating the control method of the first embodiment, and a diagram explaining the user's operation and the processor's processing in manual mode.

[0024] Figure 3B This is a flowchart illustrating the control method of the first embodiment, and a diagram explaining the processor's processing in autonomous mode.

[0025] Figure 4AThis is a diagram illustrating the operation of the endoscope in the steps of determining the first position information and the first rotation angle information.

[0026] Figure 4B This diagram illustrates the operation of the endoscope in the steps of determining the second position information and the second rotation angle information.

[0027] Figure 5A This is a diagram showing the endoscopic image at point O.

[0028] Figure 5B This is a diagram showing the endoscopic image at point B.

[0029] Figure 5C This indicates that the vertical direction has been adjusted by rotation. Figure 5B The image is an endoscopic image.

[0030] Figure 6A This is a diagram showing the endoscopic image at point A.

[0031] Figure 6B This indicates that the vertical direction has been adjusted by rotation. Figure 6A The image is an endoscopic image.

[0032] Figure 7 This is a diagram showing the position and rotation angle information stored in the storage unit in manual mode.

[0033] Figure 8A This is a timing diagram illustrating the control method of the second embodiment, and it is a diagram explaining the user's operation and the processor's processing in manual mode.

[0034] Figure 8B This is a flowchart illustrating the control method of the second embodiment, and a diagram explaining the processor's processing in autonomous mode.

[0035] Figure 9 This is a flowchart illustrating the control method of the third embodiment, and a diagram explaining the processor's processing in autonomous mode.

[0036] Figure 10 This is a diagram showing the first modified example of a strabismus endoscope.

[0037] Figure 11A This is a timing diagram representing the control method of the first variant, and a diagram illustrating the user's operation and the processor's processing in manual mode.

[0038] Figure 11B This is a flowchart illustrating the control method of the first variant, and a diagram illustrating the processor's processing in autonomous mode.

[0039] Figure 12 This is a diagram showing an endoscope with a curved section, representing a second modified example.

[0040] Figure 13A It is a timing diagram representing the control method of other variations, and a diagram illustrating the user's operation and the processor's processing in manual mode.

[0041] Figure 13B This is a flowchart illustrating the control method of other variations, and a diagram illustrating the processor's processing in autonomous mode.

[0042] Figure 14A It means Figure 1A An external view of the overall structure of a variant of an endoscope system.

[0043] Figure 14B It means Figure 1A An appearance diagram of the overall structure of other variations of the endoscope system. Detailed Implementation

[0044] (First Implementation)

[0045] The endoscope system, control device, control method, and recording medium of the first embodiment of the present invention will be described with reference to the accompanying drawings.

[0046] like Figure 1A As shown, the endoscope system 10 of this embodiment is used to insert an endoscope 2 and one or more treatment devices 6 into the body of a patient X who is the subject of the examination, and to perform surgery by observing the treatment devices 6 with the endoscope 2 while treating the treatment site with the treatment devices 6, for example, for laparoscopic surgery.

[0047] like Figure 1B As shown, endoscope 2 is inserted into the body of the patient, such as the abdominal cavity, through an opening H formed in the body wall. Thus, endoscope 2 is fixed relative to the patient, supported by the body wall at the position of the opening H, which serves as a pivot point, and can swing about a pivot (first pivot) P1 passing through the pivot point H. Figure 1A and Figure 1B In the laparoscopic surgery shown, pivot P1 extends in the anteroposterior direction from the ventral side to the dorsal side of the patient X. By oscillating the endoscope 2 around pivot P1, the imaging area captured by the endoscope 2 can move between a first region containing the aorta F and a second region containing the pelvis G.

[0048] Endoscope 2 and treatment device 6 can also be inserted into the patient's body through a cannula that extends through orifice H. The cannula is a cylindrical instrument with openings at both ends. In this case, endoscope 2 is supported by the cannula at orifice H.

[0049] like Figure 1A as well as Figure 2As shown, the endoscope system 10 includes: an endoscope 2; a moving device 3 that holds the endoscope 2 and moves the endoscope 2 within the body being examined; an endoscope processor 4 that is connected to the endoscope 2 and processes the endoscopic image E captured by the endoscope 2; a control device 1 that is connected to the moving device 3 and the endoscope processor 4 and controls the moving device 3; and a display device 5 that is connected to the endoscope processor 4 and displays the endoscopic image E.

[0050] Endoscope 2 is a direct-viewing endoscope having a visual axis (optical axis) C coaxial with the length axis I of endoscope 2, such as a rigid endoscope. Endoscope 2 has an imaging element 2a that captures images of the patient X, for example, inside the abdominal cavity, to obtain an endoscopic image E including the tip of the treatment instrument 6 (see reference). Figures 5A to 6B The imaging element 2a is, for example, a three-dimensional camera disposed at the front end of the endoscope 2, which captures a stereoscopic image as the endoscope image E. The imaging element 2a is an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, which converts light received from a specified area into an electrical signal through photoelectric conversion, thereby generating an image of the specified area. The stereoscopic image as the endoscope image E is generated by image processing two images with parallax, such as the endoscope processor 4. In this case, the front end of the endoscope 2 has a stereoscopic optical system.

[0051] The endoscopic image E is sent from the endoscope 2 to the endoscope processor 4, where necessary processing is performed. The image is then sent from the endoscope processor 4 to the display device 5 and displayed on the display screen 5a. The display device 5 can be any display such as a liquid crystal display (LCD) or an organic EL display. The surgeon observes the endoscopic image E displayed on the display screen 5a while operating the surgical instrument 6 inserted into the body. The display device 5 may also be equipped with a speaker or other sound device.

[0052] In addition to the display device 5, a terminal can also be provided for user use, communicating with the control device 1 and the endoscope processor 4 via a communication network, and displaying the endoscopic image E on the terminal. The terminal is not particularly limited and can be a laptop computer, tablet computer, or smartphone, etc.

[0053] The moving device 3 includes a robotic arm 3a (including an electric endoscope holder) that holds the endoscope 2 and performs three-dimensional control on the position and orientation of the endoscope 2. The moving device 3 has multiple joints 3b and 3c, and through the movement of the multiple joints 3b and 3c, the endoscope 2 can be moved around the pivot P1 to change the position and orientation of the endoscope 2 in three dimensions.

[0054] like Figure 1C As shown, joint 3c is a rotary joint that allows the endoscope 2 to rotate about the length axis I, and is provided, for example, at the front end of the robotic arm 3a. By rotating joint 3c, the endoscope 2 rotates about the optical axis C, which is coaxial with the length axis I. As a result, the rotation angle of the subject in the endoscopic image E, that is, the vertical direction of the endoscopic image E, changes.

[0055] The mobile device 3 is equipped with multiple angle sensors 3d that detect the rotation angle of each of the multiple joints 3b and 3c. The angle sensors 3d are, for example, encoders, potentiometers or Hall sensors installed on each of the joints 3b and 3c.

[0056] like Figure 2 As shown, the control device 1 includes at least one processor 11 (similar to a central processing unit), a memory 12, a storage unit 13, an input interface 14, an output interface 15, and a user interface 16. The control device 1 can be, for example, a desktop computer, a tablet computer, a laptop computer, a smartphone, or a mobile phone.

[0057] The processor 11 can be a single-processor, multi-processor, or multi-core processor. The processor 11 reads and executes the program stored in the storage unit 13.

[0058] The memory 12 is, for example, a semiconductor memory containing ROM (read-only memory) or RAM (random access memory) regions. The memory 12 may also store data required for processing by the processor 11 in the same way as the storage unit 13 described later (i.e., it may also operate as a "storage unit").

[0059] Storage unit 13 is a computer-readable non-transitory recording medium, such as a non-volatile recording medium including semiconductor memory like hard disks or flash memory. Storage unit 13 stores various programs including a follow-up control program (not shown) and an image control program (control program) 1a, as well as data required for processing by processor 11. Part of the processing executed by processor 11, described later, can also be implemented using dedicated logic circuits and hardware such as FPGA (Field Programmable Gate Array), SoC (System-on-a-Chip), ASIC (Application Specific Integrated Circuit), and PLD (Programmable Logic Device).

[0060] Storage unit 13 can also be a server, such as a cloud server, that is connected to control device 1 via a communication network and has a communication interface, instead of a recording medium built into control device 1. The communication network can be, for example, a public line such as the Internet, a private line, or a LAN (Local Area Network). The connection between various devices can be either wired or wireless.

[0061] Alternatively, the endoscope processor 4 that processes the endoscopic image E may also include a processor 11. That is, similar to the processor 11 included in the control device 1, the endoscope processor 4 may include a processor, dedicated logic circuitry, or hardware, and perform the processing described later in the same manner as the processor 11. The endoscope processor 4 and the control device 1 may also be integrated. Alternatively, one or more processors may be provided in both the endoscope processor 4 and the control device 1.

[0062] The structure of the control device 1, which includes at least one processor 11, memory 12, storage unit 13, input interface 14, output interface 15, and user interface 16, can also be separated from the endoscope processor 4 and the control device 1, and further disposed in the terminal used by the user. Alternatively, the control device 1 can also be integrated with the mobile device 3.

[0063] Input interface 14 and output interface 15 are connected to endoscope processor 4. Control device 1 can acquire endoscopic image E from endoscope 2 via endoscope processor 4 and output endoscopic image E to display device 5 via endoscope processor 4. Alternatively, input interface 14 can be directly connected to endoscope 2, and output interface 15 can be directly connected to display device 5, allowing control device 1 to directly acquire endoscopic image E from endoscope 2 and directly output endoscopic image E to display device 5.

[0064] Additionally, input interface 14 and output interface 15 are connected to the mobile device 3. The control device 1 obtains information about the rotation angles of joints 3b and 3c detected by angle sensor 3d from the mobile device 3, and sends control signals for driving joints 3b and 3c to the mobile device 3.

[0065] User interface 16 has input devices such as buttons, mouse, keyboard and touch panel for users such as surgeons to input into user interface 16, and accepts input from users.

[0066] Additionally, the user interface 16 has a unit, such as a switch, for allowing the user to switch between the manual mode and the autonomous mode, as described later.

[0067] Furthermore, the user interface 16 is configured to accept a first instruction and a second instruction from the user. The first instruction and the second instruction are instructions for causing the control device 1 to register the position information and rotation angle information described later. For example, the user interface 16 has a button operated by the operator, which accepts the first instruction by being operated on the button for the first time, and accepts the second instruction by being operated on the button for the second time.

[0068] The processor 11 can operate in either manual or autonomous mode.

[0069] The manual mode allows the surgeon or other user to operate the endoscope 2. In manual mode, the surgeon can manually move the endoscope 2 by holding its base. Alternatively, the surgeon can remotely operate the endoscope 2 using an operating device connected to the moving device 3. The operating device may include buttons, joysticks, and touch panels.

[0070] In autonomous mode, the moving device 3 is controlled based on the position of the instrument 6 projected within the endoscopic image E, thereby enabling the endoscope 2 to automatically follow the instrument 6. In autonomous mode, the processor 11 obtains the three-dimensional position of the tip of the instrument 6 based on the endoscopic image E, and controls the moving device 3 based on the three-dimensional position of the tip of the instrument 6 and the three-dimensional position of a predetermined target point set within the field of view of the endoscope 2. The target point is, for example, a point on the optical axis C corresponding to the center point of the endoscopic image E. Thus, the control device 1 controls the movement of the endoscope 2, causing the endoscope 2 to follow the instrument 6 so that the tip of the instrument 6 is positioned at the center point within the endoscopic image E.

[0071] Furthermore, in autonomous mode, the processor 11 executes the image control program 1a read into the memory 12. Figure 3A as well as Figure 3B The control method shown controls the rotation angle of the endoscopic image E displayed on the display screen 5a.

[0072] Next, the control method executed by the processor 11 will be described.

[0073] like Figure 3A and Figure 3BAs shown, the control method of this embodiment includes: step SB2, setting the initial position of the endoscope 2; steps SB3 and SB4, determining the first position information and the first rotation angle information of the first region inside the subject; steps SB5 and SB6, determining the second position information and the second rotation angle information of the second region inside the subject; steps SB7 and SB8, calculating the third position information and the third rotation angle information of the third region inside the subject; step SB9, storing the position information and rotation angle information in the storage unit 13; steps SC4 to SC9, rotating the endoscope image E according to the current imaging area captured by the endoscope 2; and step SC10, outputting the rotated endoscope image E to the display device 5.

[0074] like Figure 3A As shown, steps SB2 to SB9 are executed in manual mode, as follows: Figure 3B As shown, steps SC3 to SC10 are executed in autonomous mode.

[0075] After inserting the endoscope 2, held in the moving device 3, into the abdominal cavity, the surgeon or other user switches to manual mode (SA1, SB1) and begins a panoramic view (SA3) by moving the endoscope 2 within the abdominal cavity. Panoramic view is the process of observing the entire abdominal cavity to confirm the location of organs and tissues. Since the location of organs and tissues varies from patient to patient, this process is required for each insertion. During panoramic view, the surgeon rotates the endoscope 2 around pivot P1, thereby observing an area including at least two specific tissues with anatomical features. In this embodiment, the specific tissues are the aorta F and the pelvis G.

[0076] like Figure 3A As shown, before panoramic viewing, the surgeon registers the initial position of endoscope 2 with the control device 1 (SA2). For example, the surgeon positions endoscope 2 in the desired initial position and operates a designated button on the user interface 16. In response to the operation of the designated button, the processor 11 calculates the current position φ of endoscope 2, sets the current position φ to the initial position φ = 0°, and stores it in the storage unit 13 (SB2). Position φ is the circumferential position of endoscope 2 about pivot P1, calculated based on the rotation angles of joints 3b and 3c detected by angle sensor 3d. Position φ represents the circumferential position of the imaging area about pivot P1.

[0077] Next, as Figure 4A and Figure 5A As shown, the surgeon positions endoscope 2 at point O when taking a frontal view of the aorta F, and adjusts the rotation angle ω of endoscope 2 around the optical axis C so that the aorta F is positioned at the desired rotation angle (SA4) within the endoscopic image E. Here, the rotation angle of the aorta F is its circumferential position around the center point of the endoscopic image E. In this embodiment, as... Figure 5A As shown, the rotation angle ω is adjusted so that the aorta F is positioned horizontally within the endoscopic image E. Next, the surgeon inputs a first instruction (SA5) into the user interface 16.

[0078] After the first instruction is input, the surgeon maintains the adjusted rotation angle ω at point O while rotating endoscope 2 around pivot P1 from point O, thereby observing the entire aorta F using endoscope 2. For example... Figure 5A and Figure 5B As shown, as endoscope 2 rotates from point O towards point B, the aorta F rotates and moves within the endoscopic image E. Point B is the endpoint of the area of ​​the aorta F observed within the endoscopic image E.

[0079] In response to the user interface 16 receiving the first instruction, the processor 11 determines, based on the endoscopic image E, first location information and first rotation angle information (SB3, SB4) of a first region containing the aorta (first specific tissue) F. The first rotation angle information is information that defines the rotation angle of the endoscopic image E of the first region.

[0080] Specifically, the storage unit 13 stores a learned model 1b obtained by machine learning of the correspondence between images containing specific tissues and types of specific tissues. In step SB3, the processor 11 uses the learned model 1b to identify the aorta F in the endoscopic image E, and determines the range of the position φ of the endoscope 2 within the endoscopic image E containing the aorta F as the first position information. That is, the first region is the region between point O and point B.

[0081] For example, the first position information is φ = 0° to 20°. Thus, instead of setting the initial position in steps SA2 and SB2, the processor 11 can also set the position φ of the endoscope 2 at the moment of receiving the first instruction to the initial position φ = 0°. That is, the initial position is determined at the time and location desired by the user.

[0082] Furthermore, the processor 11 can also set the position φ of the endoscope 2 at the moment of receiving the first instruction as the first position information without going through the processing of the learned model 1b. That is, the first position information is determined at the time and location desired by the user.

[0083] Next, in step SB4, the processor 11 sets the endoscope image E at the moment when the user interface 16 receives the first instruction and the rotation angle ω of the endoscope 2 as the first reference endoscope image and the first reference rotation angle, respectively, and determines the first rotation angle information based on the first reference endoscope image and the first reference rotation angle.

[0084] Specifically, the processor 11 calculates a first reference rotation angle relative to a predetermined initial rotation angle ω = 0°, which serves as the target rotation angle θt of the endoscopic image E at the position φ at the moment the first instruction is received. The calculated target rotation angle θt represents the amount of rotation of the endoscopic image E required to horizontally position the aorta F within the endoscopic image E at the position φ at the moment the first instruction is received. In this embodiment, the first reference rotation angle ω is set to the initial rotation angle 0°.

[0085] Next, processor 11 calculates the rotation amount Δθ of the endoscopic image E required to align the aorta F in the endoscopic image E with the aorta F in the first reference endoscopic image for the endoscopic image E acquired at other positions φ included in the first position information. Then, processor 11 calculates the target rotation angle θt at the other positions φ by adding the rotation amount Δθ to the first reference rotation angle. The calculated target rotation angle θt represents the rotation amount of the endoscopic image E required to horizontally position the aorta F within the endoscopic image E at the other positions φ. Figure 5C This indicates the result after rotating the target angle θt at point B. Figure 5B Endoscopic image E.

[0086] As described above, processor 11 calculates the target rotation angle θt for the endoscopic image E in which the aorta F is horizontally positioned at each position φ = 0°, ..., 20° included in the first position information, and determines the target rotation angle θt at each position φ = 0°, ..., 20° as the first rotation angle information. Figure 7 In the first rotation angle information, only the target rotation angles θt = 0° and -10° at φ = 0° and 20° are recorded representatively.

[0087] Next, as Figure 4B As shown, the surgeon positions endoscope 2 at the location (point D) where the pelvis G is to be photographed. When observing the pelvis G with an initial rotation angle ω = 0°, as... Figure 6A As shown, the pelvis G can be positioned improperly within the endoscopic image E. The surgeon adjusts the rotation angle ω of the endoscope 2 about the optical axis C to position the pelvis G at the desired rotation angle within the endoscopic image E (SA6), and inputs a second instruction (SA7) to the user interface 16. In this embodiment, as... Figure 6B As shown, adjust the rotation angle ω so that the pelvis G is positioned above in the endoscopic image E.

[0088] After inputting the second instruction, the surgeon maintains the rotation angle ω adjusted at point D while rotating endoscope 2 around pivot P1 from point D, thereby observing the entire pelvis G using endoscope 2. At this time, as endoscope 2 rotates from point D toward point A, pelvis G also rotates and moves within endoscopic image E. Point A is the endpoint of the area of ​​pelvis G observed within endoscopic image E.

[0089] In response to the user interface 16 receiving the second instruction, the processor 11 determines second position information and second rotation angle information (SB5, SB6) of the second region containing the pelvis (second specific tissue) G based on the endoscopic image E. The second rotation angle information is information that specifies the rotation angle of the endoscopic image E of the second region.

[0090] Specifically, in step SB5, the processor 11 uses the learned model 1b to identify the pelvis G within the endoscopic image E, and determines the range of the position φ of the endoscope 2 within the endoscopic image E containing the pelvis G as the second position information. That is, the second region is the area between point D and point A. For example, the second position information is φ = 70° to 90°.

[0091] Furthermore, regarding the second position information, the processor 11 may also set the position φ of the endoscope 2 at the moment of receiving the second instruction as the second position information without going through the processing of the learned model 1b. That is, the second position information is determined at the time and location desired by the user.

[0092] Next, in step SB6, the processor 11 sets the endoscope image E at the moment when the user interface 16 receives the second instruction and the rotation angle ω of the endoscope 2 as the second reference endoscope image and the second reference rotation angle, respectively, and determines the second rotation angle information based on the second reference endoscope image and the second reference rotation angle.

[0093] Specifically, processor 11 calculates a second reference rotation angle relative to the initial rotation angle ω = 0°, which is the target rotation angle θt of the endoscopic image E at position φ when the second instruction is received. The calculated target rotation angle θt represents the amount of rotation required to position the pelvis G above the endoscopic image E within the endoscopic image E at position φ when the second instruction is received.

[0094] Next, the processor 11 calculates the rotation amount Δθ of the endoscopic image E required to align the pelvis G in the endoscopic image E with the pelvis G in the second reference endoscopic image, taken from other positions φ included in the second position information. Then, the processor 11 calculates the target rotation angle θt at the other positions φ by adding the rotation amount Δθ to the second reference rotation angle. The calculated target rotation angle θt represents the rotation amount of the endoscopic image E required to position the pelvis G superiorly within the endoscopic image E at the other positions φ.

[0095] As described above, processor 11 calculates the target rotation angle θt for positioning the pelvis G above the endoscopic image E at each position φ = 70°, ..., 90° included in the second position information, and determines the target rotation angle θt at each position φ = 70°, ..., 90° as the second rotation angle information. Figure 7 In the second rotation angle information, only the target rotation angles θt = 100° and 90° at φ = 70° and 90° are recorded representatively.

[0096] Next, the processor 11 calculates the third position information and the third rotation angle information (SB7, SB8) of the third region based on the first position information, the first rotation angle information, the second position information, and the second rotation angle information. The third region is a region different from the first region and the second region, and in this embodiment, it is the region between point A and point B.

[0097] In step SB7, the processor 11 determines the range of position φ between the first position information and the second position information as the third position information. For example, the third position information is φ = 20° to 70°.

[0098] Next, in step SB8, the processor 11 calculates the third rotation angle information based on the first position information, the second position information, the third position information, the first rotation angle information, and the second rotation angle information. The third rotation angle information is information that defines the rotation angle of the endoscopic image E of the third region.

[0099] Specifically, the processor 11 calculates the positional relationship between the third positional information and the first and second positional information, and calculates the third rotational angle information based on the positional relationship, the first rotational angle information, and the second rotational angle information.

[0100] As an example, each position φ (point M) in the third position information is considered to be an inner dividing point that divides the trajectory between points A and B into a ratio of m:n. The processor 11 calculates the target rotation angle θt at each position φ based on the ratio m:n, the rotation angle of point A (100°), and the rotation angle of point B (-10°). For example, if position φ = 45° divides the trajectory between points A and B into a 1:1 ratio, then the target rotation angle θt at position φ = 45° is the midpoint between -10° and 100°, i.e., 45°.

[0101] Therefore, the target rotation angle θt is calculated as the position φ changes from point B towards point A, gradually changing from 100° to -10°.

[0102] Processor 11 determines the target rotation angle θt at each position φ = 20°, ..., 70° as the third rotation angle information. Figure 7 In the middle, as the third rotation angle information, only the target rotation angle θt = 45° at φ = 45° is recorded representatively.

[0103] That is, the third region is a region in which specific tissues, such as the pelvis G and aorta F in the first and second regions, are not reflected in the endoscopic image, which are parameters of the rotation angle of the endoscopic image E. In such regions, the identification of specific tissues by the learned model 1b and the determination of the desired rotation angle by the user are difficult. Therefore, it is necessary to calculate the third position information and the third rotation angle information based on the first and second position information and the first and second rotation angle information of the first and second regions.

[0104] Next, in step SB9, the processor 11 stores the first position information, first rotation angle information, second position information, second rotation angle information, third position information, and third rotation angle information determined in steps SB3 to SB8 in the storage unit 13. Thus, as... Figure 7 As shown, the storage unit 13 generates data including the rotation angle φ of the endoscope 2, which indicates the position of the imaging area, and the target rotation angle θt of the endoscope image E at each rotation angle φ.

[0105] After completing the panoramic view, the surgeon switched from manual to autonomous mode and used instrument 6 to treat the major artery F and the pelvis G. For example... Figure 3B As shown, when the surgeon switches to autonomous mode (SC2), the processor 11 rotates the rotary joint 3c to make the rotation angle ω of the endoscope 2 match the initial rotation angle 0°. While maintaining the rotation angle ω at 0°, the processor 11 controls the movement device 3 to make the endoscope 2 follow the tip of the treatment instrument 6 (SC3). Furthermore, the processor 11 and the endoscope 2 follow each other in parallel to control the vertical direction of the endoscopic image E displayed on the display screen 5a (SC4-SC10).

[0106] During the startup of devices 1 and 3, processor 11 receives the rotation angles of joints 3b and 3c sequentially from mobile device 3, and calculates the current position φ(SC1) of endoscope 2 based on the rotation angles of joints 3b and 3c.

[0107] The processor 11 determines which of the first, second, and third regions (SC4, SC6, SC8) the current camera area is contained in based on the current position of the endoscope 2, the first position information, and the second position information.

[0108] Specifically, if the current position φ is included in the first position information (φ = 0° to 20°), the processor 11 determines that the current imaging area is included in the first area ("Yes" in SC4). Next, the processor 11 rotates the endoscope image E within the plane of the endoscope image E according to the first rotation angle information stored in the storage unit 13 (SC5). Specifically, the processor 11 reads the target rotation angle θt of the current position φ from the storage unit 13 and rotates the endoscope image E by the target rotation angle θt through image processing. Then, the processor 11 outputs the rotated endoscope image E from the control device 1 to the display device 5 and displays it on the display screen 5a (SC10).

[0109] In the rotated endoscopic image E, the aorta F is positioned horizontally. Therefore, during the process of obtaining the endoscopic image E containing the aorta F by moving the endoscope 2 within the range of φ = 0° to 20°, the aorta F displayed in the endoscopic image E on the display screen 5a is maintained horizontally. For example, if the endoscope 2 swings 20° around the pivot P1 from point O to point B, the endoscopic image E rotates from 0° to -10°.

[0110] If the current position φ is included in the second position information (φ = 70° to 90°), the processor 11 determines that the current imaging area is included in the second area ("No" in SC4 and "Yes" in SC6). Next, the processor 11 rotates the endoscope image E in the plane of the endoscope image E according to the second rotation angle information stored in the storage unit 13 (SC7). Specifically, the processor 11 reads the target rotation angle θt of the current position φ from the storage unit 13 and rotates the endoscope image E by the target rotation angle θt through image processing. Then, the processor 11 outputs the rotated endoscope image E from the control device 1 to the display device 5 and displays it on the display screen 5a (SC10).

[0111] In the rotated endoscopic image E, the pelvis G is positioned upwards. Therefore, during the period when the endoscope 2 moves within the range of φ = 70° to 90° to obtain the endoscopic image E containing the pelvis G, the pelvis G displayed in the endoscopic image E on the display screen 5a remains upwards. For example, if the endoscope 2 swings 20° around the pivot P1 from point A to point D, the endoscopic image E rotates from 100° to 90°.

[0112] If the current position φ is not included in either the first position information or the second position information ("No" in SC4 and "No" in SC6), the processor 11 determines that the current imaging area is included in the third area (SC8). Next, the processor 11 rotates the endoscope image E within the plane of the endoscope image E according to the third rotation angle information stored in the storage unit 13 (SC9). Specifically, the processor 11 reads the rotation angle of the current position φ from the storage unit 13 and rotates the endoscope image E by the rotation angle through image processing. Then, the processor 11 outputs the rotated endoscope image E from the control device 1 to the display device 5 and displays it on the display screen 5a (SC10).

[0113] The endoscopic image E displayed on display screen 5a rotates by a target rotation angle θt corresponding to the position φ. As the position φ changes from the first region side to the second region side, the target rotation angle θt gradually changes from the target rotation angle of the first region to the target rotation angle of the second region. Therefore, for example, when the endoscope 2 swings around the pivot P1 from point B to point A, the endoscopic image E displayed on display screen 5a rotates from -10° to 100° in one direction.

[0114] As explained above, according to this embodiment, first position information and first rotation angle information of a first region containing a specific tissue F are stored in the storage unit 13. The first rotation angle information specifies a target rotation angle θt for positioning the specific tissue F at the endoscopic image E at the rotation angle desired by the surgeon. Additionally, second position information and second rotation angle information of a second region containing a specific tissue G are stored in the storage unit 13. The second rotation angle information specifies a target rotation angle θt for positioning the specific tissue G at the rotation angle desired by the surgeon. Furthermore, a third rotation angle information for a third region between the first and second regions is interpolated between the target rotation angle θt of the first rotation angle information and the target rotation angle θt of the second rotation angle information, and is stored in the storage unit 13.

[0115] Subsequently, in autonomous mode, the vertical orientation of the endoscopic image E is automatically adjusted by rotating it by a target rotation angle θt corresponding to the current position φ of the imaging area. Specifically, when the current imaging area is a first or second region containing specific tissues F and G, the endoscopic image E is automatically rotated to a target rotation angle θt corresponding to the specific tissues F and G with a predetermined rotation angle. When the current imaging area is a third region that does not contain specific tissues F and G, the endoscopic image E is automatically rotated to an appropriate target rotation angle θt estimated based on the first and second rotation angle information.

[0116] This allows the operator to be provided with an endoscopic image E in the appropriate vertical direction corresponding to the current position of the imaging area within the abdominal cavity.

[0117] Furthermore, by automatically adjusting the vertical orientation of the endoscopic image E, the surgeon's stress can be reduced, and the procedure time can be shortened. That is, when the surgeon manually adjusts the vertical orientation of the endoscopic image E, the surgeon must temporarily release their hand from the operating instrument 6 and manually rotate the endoscope 2. According to this embodiment, the surgeon does not need to operate the endoscope 2 for vertical adjustment, thus allowing the surgeon to continue the procedure without interruption.

[0118] (Second Implementation)

[0119] Next, the endoscope system, control device, control method, and recording medium of the second embodiment of the present invention will be described with reference to the accompanying drawings.

[0120] This embodiment differs from the first embodiment in that the processor 11 rotates the endoscopic image E by rotating the endoscope 2 instead of performing image processing. In this embodiment, structures that differ from those in the first embodiment will be described, while structures identical to those in the first embodiment will be labeled with the same reference numerals and their descriptions will be omitted.

[0121] The endoscope system 10 of this embodiment includes, like the first embodiment, a control device 1, an endoscope 2, a moving device 3, an endoscope processor 4, and a display device 5.

[0122] Figure 8A as well as Figure 8B This indicates the control method executed by the processor 11 in this embodiment.

[0123] like Figure 8A and Figure 8B As shown, the control method of this embodiment includes: step SB2, determining the initial position of the endoscope 2; steps SB3 and SB4', determining the first position information and the first rotation angle information of the first region inside the subject; steps SB5 and SB6', determining the second position information and the second rotation angle information of the second region inside the subject; steps SB7 and SB8', calculating the third position information and the third rotation angle information of the third region inside the subject; step SB9, storing the position information and rotation angle information in the storage unit 13; steps SC4 to SC9', rotating the endoscope image E according to the current imaging area captured by the endoscope 2; and step SC10, outputting the rotated endoscope image E to the display device 5.

[0124] like Figure 8A As shown, steps SB2 to SB9 are executed in manual mode, as follows: Figure 8BAs shown, steps SC4 to SC10 are executed in autonomous mode.

[0125] Similar to the first embodiment, the user performs steps SA1 to SA5. In response to the user interface 16 receiving the first instruction, the processor 11 determines the first position information and the first rotation angle information (SB3, SB4') of the first region based on the endoscopic image E.

[0126] Specifically, in the next step SB4' after step SB3, the processor 11 sets the endoscope image E at the moment when the user interface 16 receives the first instruction and the rotation angle ω of the endoscope 2 as the first reference endoscope image and the first reference rotation angle, respectively.

[0127] Next, the processor 11 calculates a first reference rotation angle relative to a predetermined initial rotation angle ω = 0°, which is the target rotation angle ωt of the endoscope 2 at the position φ at the moment of receiving the first instruction.

[0128] Next, the processor 11 calculates the rotation amount Δθ of the endoscope image E acquired at other positions φ included in the first position information, which is required to make the aorta F in the endoscope image E consistent with the aorta F in the first reference endoscope image. Then, the processor 11 calculates the target rotation angle ωt of the endoscope 2 at other positions φ by adding the rotation amount Δθ to the first reference rotation angle.

[0129] As described above, the processor 11 calculates the target rotation angle ωt of the endoscope 2 for horizontally positioning the aorta F at each position φ = 0°, ..., 20° included in the first position information, and determines the target rotation angle ωt at each position φ = 0°, ..., 20° as the first rotation angle information.

[0130] Next, the user proceeds to steps SA6 and SA7. In response to the user interface 16 receiving the second instruction, the processor 11 determines the second position information and the second rotation angle information (SB5, SB6') of the second region based on the endoscopic image E.

[0131] Specifically, in the next step SB6' after step SB5, the processor 11 sets the endoscope image E at the moment when the user interface 16 accepts the second instruction and the rotation angle ω of the endoscope 2 as the second reference endoscope image and the second reference rotation angle, respectively.

[0132] Next, the processor 11 calculates a second reference rotation angle relative to the initial rotation angle ω = 0°, which is the target rotation angle ωt of the endoscope 2 at the position φ at the moment of receiving the second instruction.

[0133] Next, the processor 11 calculates the rotation amount Δθ of the endoscope image E acquired at other positions φ included in the second position information, which is required to make the pelvis G in the endoscope image E consistent with the aorta G in the second reference endoscope image. Then, the processor 11 calculates the target rotation angle ωt of the endoscope 2 at other positions φ by adding the rotation amount Δθ to the second reference rotation angle.

[0134] As described above, the processor 11 calculates the target rotation angle ωt for positioning the pelvis G on top of the endoscope 2 at each position φ = 70°, ..., 90° included in the second position information, and determines the target rotation angle ωt at each position φ = 70°, ..., 90° as the second rotation angle information.

[0135] Next, the processor 11 calculates the third position information and the third rotation angle information of the third region based on the first position information, the first rotation angle information, the second position information, and the second rotation angle information (SB7, SB8'). Specifically, in the next step SB8' after step SB7, the processor 11, similarly to step SB8', determines the target rotation angle ωt at each position φ = 20°, ..., 70° of the third position information as the third rotation angle information.

[0136] Next, in step SB9, the processor 11 causes the storage unit 13 to store the position information and rotation angle information determined in steps SB3, SB4', SB5, SB6', SB7, and SB8'. As a result, data is generated in the storage unit 13 that includes the rotation angle φ of the endoscope 2 representing the position of the imaging area and the target rotation angle ωt of the endoscope 2 at each rotation angle φ.

[0137] Next, as Figure 8B As shown, processor 11 calculates the current position φ of endoscope 2 (SC1). When switching to autonomous mode ("Yes" in SC2), processor 11 determines which of the first, second, and third regions the current camera area is contained in (SC4, SC6, SC8).

[0138] If it is determined that the current camera area is contained within the first area ("Yes" in SC4), the processor 11 rotates the endoscope 2 according to the first rotation angle information stored in the storage unit 13 (SC5'). Specifically, the processor 11 reads the target rotation angle ωt of the current position φ from the storage unit 13, and rotates the endoscope image E by rotating the endoscope 2 to the target rotation angle ωt.

[0139] If it is determined that the current camera area is contained within the second area (SC4 "No" and SC6 "Yes"), the processor 11 rotates the endoscope 2 according to the second rotation angle information stored in the storage unit 13 (SC7'). Specifically, the processor 11 reads the target rotation angle ωt of the current position φ from the storage unit 13, and rotates the endoscope image E by rotating the endoscope 2 by the target rotation angle ωt.

[0140] If it is determined that the current camera area is contained within the third area (SC7), the processor 11 rotates the endoscope 2 according to the third rotation angle information stored in the storage unit 13 (SC8'). Specifically, the processor 11 reads the target rotation angle ωt of the current position φ from the storage unit 13, and rotates the endoscope image E by rotating the endoscope 2 by the target rotation angle ωt.

[0141] After step SC5', SC7' or SC9', the processor 11 outputs the rotated endoscope image E from the control device 1 to the display device 5 and displays it on the display screen 5a (SC10).

[0142] As explained above, according to this embodiment, similarly to the first embodiment, in autonomous mode, the endoscope 2 is rotated to a target rotation angle ωt corresponding to the current imaging region position φ, thereby automatically adjusting the vertical direction of the endoscopic image E. Specifically, when the current imaging region is a first or second region containing specific tissues F and G, the endoscope 2 is automatically rotated to a target rotation angle ωt that positions the specific tissues F and G at a predetermined rotation angle. When the current imaging region is a third region that does not contain specific tissues F and G, the endoscope 2 is automatically rotated to an appropriate target rotation angle ωt estimated based on the first and second rotation angle information.

[0143] This allows the operator to be provided with an endoscopic image E in the appropriate vertical direction corresponding to the current location of the imaging area within the abdominal cavity. Furthermore, by automatically adjusting the vertical direction of the endoscopic image E, operator stress can be reduced, and procedure time can be shortened.

[0144] Furthermore, according to this embodiment, the endoscope image E is rotated by rotating the endoscope 2 around the optical axis C, thereby eliminating image processing required to rotate the endoscope image E and reducing the load on the processor 11. In addition, the user can visually confirm the rotation angle ω of the portion of the endoscope 2 disposed outside the body, and can intuitively identify the vertical direction of the endoscope image E.

[0145] In this embodiment, the endoscope image E is rotated by rotating the endoscope 2 as a whole around the optical axis C. However, it is also possible to rotate the imaging element 2a around the optical axis C while maintaining the rotation angle ω of the endoscope 2 around the optical axis C. In this case, a rotation mechanism for rotating the imaging element 2a is provided in the endoscope 2.

[0146] By rotating the imaging element 2a relative to the body of the endoscope 2, the endoscope image E can also be rotated in the same way as when the endoscope 2 as a whole is rotated.

[0147] (Third Implementation)

[0148] Next, the endoscope system, control device, control method, and recording medium of the third embodiment of the present invention will be described with reference to the accompanying drawings.

[0149] This embodiment differs from the first and second embodiments in that it rotates the endoscopic image E by combining the rotation of the endoscope 2 around the optical axis C with image processing. In this embodiment, structures that differ from the first and second embodiments will be described, while structures that are the same as those in the first and second embodiments will be labeled with the same reference numerals and their descriptions will be omitted.

[0150] The endoscope system 10 of this embodiment includes, like the first embodiment, a control device 1, an endoscope 2, a moving device 3, an endoscope processor 4, and a display device 5.

[0151] Figure 9 This indicates the control method executed by the processor 11 in autonomous mode in this embodiment. In addition to steps SB2, SB3, SB4', SB5, SB6', SB7, SB8', SB9, SC1 to SC4, SC5', SC6, SC7', SC8, and SC9' described in the second embodiment, the control method of this embodiment also includes step SC11, which determines whether the rotation angle ω of the endoscope 2 is a predetermined limit angle, and step SC12, which rotates the endoscope image E through image processing.

[0152] After step SB9, as Figure 9 As shown, processor 11 calculates the current position φ (SC1) of endoscope 2. When switching to autonomous mode ("Yes" in SC2), processor 11 executes steps SC1 to SC4, SC5', SC6, SC7', SC8, and SC9'.

[0153] In steps SC5', SC7', and SC9', the processor 11 determines whether the rotation angle ω of the endoscope 2 reaches the limit angle of the rotatable range of the endoscope 2 based on the rotation angle of the rotary joint 3c detected by the angle sensor 3d (SC11). The rotatable range of the endoscope 2 is sometimes limited by physical constraints. For example, the cables inside the endoscope 2 and the moving device 3 may twist due to the rotation of the endoscope 2, so the rotatable range of the endoscope 2 is set in a way that does not produce excessive twisting.

[0154] If the endoscope 2 rotates to the target rotation angle ωt before the rotation angle ω reaches the limit angle ("No" in SC11), the processor 11 outputs the rotated endoscope image E to the display device 5 (SC10).

[0155] On the other hand, if the limit angle is reached before the rotation angle ω reaches the target rotation angle ωt ("Yes" in SC11), the processor 11 stops the rotation of the endoscope 2 at the limit angle, and then rotates the endoscope image E by an angle that is insufficient relative to the target rotation angle ωt (SC12) through image processing, and outputs the rotated endoscope image E to the display device 5 (SC10).

[0156] As explained above, according to this embodiment, by combining the rotation of the endoscope 2 around the optical axis C with image processing, it is possible to achieve the rotation of the endoscopic image E, which is difficult to achieve by rotating the endoscope 2 alone.

[0157] Other effects of this embodiment are the same as those of the first and second embodiments, so the description is omitted.

[0158] (First variation)

[0159] Next, a first variation of the endoscope system 10, control device 1, control method, and recording medium of the first to third embodiments will be described.

[0160] like Figure 10 As shown, this variant differs from the first to third embodiments described above in that the endoscope 2 is of the oblique viewing type.

[0161] The strabismus endoscope 2 includes: an elongated insertion section 2b having a length axis I and inserted into the subject body; and an imaging section 2c containing an imaging element 2a and connected to the base of the insertion section 2b. By rotating the rotary joint 3c, the insertion section 2b and the imaging section 2c rotate integrally about the length axis I. Furthermore, in the case of a separate strabismus endoscope, the camera (imaging section 2c) and the optical observation tube (insertion section 2b) have different rotation angle information; however, in this modification, by rotating the camera and the optical observation tube integrally, they are processed as common rotation angle information.

[0162] In the case of the direct-viewing endoscope 2, the visual axis (optical axis) C is coaxial with the length axis I, so even if the endoscope 2 rotates around the length axis I, the position of the visual axis C can be maintained. On the other hand, in the case of the oblique-viewing endoscope 2, the visual axis C is tilted relative to the length axis I, so as the endoscope 2 rotates around the length axis I, the visual axis C rotates and moves around the length axis I, and the imaging area moves.

[0163] Figure 11A as well as Figure 11B This indicates the control method executed by processor 11 in this variant example. For example... Figure 11A and Figure 11B As shown, the control method of this modified example includes steps SB2', SB3 to SB9 and steps SC3', SC4 to SC10.

[0164] In step SB2', the processor 11 sets the current position φ of the endoscope 2 to the initial position φ = 0°, and sets the current attitude ω of the endoscope 2 to the initial attitude ω = 0°. The attitude ω of the endoscope 2 is the rotation angle about the length axis I, which is equivalent to the attitude of the visual axis C relative to the length axis I.

[0165] Next, in response to the user interface 16 receiving the first instruction (SA5), the processor 11 determines the first position information and the first rotation angle information (SB3, SB4), and then maintains the information of the first posture of the endoscope 2 at the moment the first instruction is received.

[0166] Next, in response to the user interface 16 receiving the second instruction (SA7), the processor 11 determines the second position information and the second rotation angle information (SB5, SB6), and then maintains the information of the second posture of the endoscope 2 at the moment the second instruction is received.

[0167] In step SB9, in addition to the position information and rotation angle information, the processor 11 also stores the first posture and the second posture in the storage unit 13. As a result, the storage unit 13 generates the following data, which includes the rotation angle φ of the endoscope 2 indicating the position of the imaging area, the target rotation angle θt of the endoscope image E at each rotation angle φ, and the first posture and the second posture of the endoscope 2 corresponding to each imaging area.

[0168] Next, in autonomous mode, the processor 11 controls the position and orientation of the endoscope 2 by controlling the moving device 3, so that the endoscope 2 follows the tip (SC3') of the treatment device 6. Here, the processor 11 controls the position and orientation of the endoscope 2 according to the first position information and the second position information, as well as the first orientation and the second orientation stored in the storage unit 13, thereby controlling the orientation ω of the endoscope 2 to the first orientation when the camera area is included in the first area, and controlling the orientation ω of the endoscope 2 to the second orientation when the camera area is included in the second area.

[0169] Then, similarly to the first embodiment, the processor 11 rotates the endoscope image E by a target rotation angle θt (SC4~SC9) according to the current camera area through image processing.

[0170] As described above, in the case of the oblique endoscope 2, the imaging area moves by rotating the endoscope 2 around the length axis I. Therefore, the control method of the second embodiment, which only uses the rotation of the endoscope 2 to rotate the endoscope image E, is difficult to control the vertical direction of the endoscope image E.

[0171] According to this variation, in manual mode, the first posture of the endoscope 2 when capturing the first region and the second posture of the endoscope 2 when capturing the second region are stored. Furthermore, in autonomous mode, when capturing the first region, the posture of the endoscope 2 is controlled to the first posture, and the vertical orientation of the endoscope image E is adjusted by rotation based on image processing. Similarly, in autonomous mode, when capturing the second region, the posture of the endoscope 2 is controlled to the second posture, and the vertical orientation of the endoscope image E is adjusted by rotation based on image processing. Thus, the vertical orientation of the endoscope image E acquired by the oblique endoscope 2 can be appropriately controlled.

[0172] (Second variation)

[0173] Next, a second variation of the endoscope system 10, control device 1, control method, and recording medium 13 of the first to third embodiments will be described.

[0174] like Figure 12 As shown, this modified example differs from the first to third embodiments described above in that the endoscope 2 has a curved portion 2d.

[0175] The endoscope 2 has: an elongated insertion portion 2b, which is inserted into the body being examined; and a curved portion 2d, which is disposed at the front end of the insertion portion 2b and is capable of bending in a direction intersecting the length axis I of the insertion portion 2b. When the curved portion 2d is bent, the visual axis C is tilted relative to the length axis I. Therefore, the visual axis C rotates and moves around the length axis I as the endoscope 2 rotates, and the imaging area moves. Furthermore, the tilt direction and tilt angle of the visual axis C relative to the length axis I vary depending on the bending direction and bending angle of the curved portion 2d.

[0176] In this modified example, the control method executed by the processor 11 is the same as in the first modified example, including steps SB2', SB3 to SB9 and steps SC3', SC4 to SC10. However, as the orientation of the endoscope 2, the rotation direction and rotation angle of the curved portion 2d are used instead of the rotation angle ω around the length axis I.

[0177] That is, in step SB2', the processor 11 sets the current bending direction and bending angle of the bent portion 2d as the initial posture. Then, in step SB9, the processor 11 stores the bending direction and bending angle of the bent portion 2d at the moment of receiving the first instruction as the first posture in the storage unit 13, and stores the bending direction and bending angle of the bent portion 2d at the moment of receiving the second instruction as the second posture in the storage unit 13.

[0178] Furthermore, in step SC3' of the autonomous mode, the processor 11 controls the position and orientation of the endoscope 2 based on the first position information and the second position information stored in the storage unit 13, as well as the first orientation and the second orientation. Thus, when the camera area is included in the first area, the bending direction and bending angle of the curved part 2d are controlled to the first orientation, and when the camera area is included in the second area, the bending direction and bending angle of the curved part 2d are controlled to the second orientation (SC3').

[0179] As described above, in the case of an endoscope 2 having a curved portion 2d, the imaging area rotates and moves by rotating the endoscope 2, depending on the bending direction and bending angle of the curved portion 2d. Therefore, it is difficult to control the vertical direction of the endoscope image E by using the control method of the second embodiment, which rotates the endoscope image E by utilizing the rotation of the endoscope 2.

[0180] According to this modified example, similarly to the first modified example, in autonomous mode, during imaging in the first region, the orientation of the endoscope 2 is controlled to a first orientation stored in manual mode, and the vertical direction of the endoscope image E is adjusted by rotation based on image processing. Furthermore, in autonomous mode, during imaging in the second region, the orientation of the endoscope 2 is controlled to a second orientation stored in manual mode, and the vertical direction of the endoscope image E is adjusted by rotation based on image processing. Thus, the vertical direction of the endoscope image E acquired by the endoscope 2 with the curved portion 2d can be appropriately controlled.

[0181] In the above embodiments and variations, the processor 11 calculates the third rotation angle information in manual mode and stores it in the storage unit 13, but this can be replaced by other methods, such as... Figure 13A as well as Figure 13B As shown, processor 11 calculates the third rotation angle information (SC13) in real time during execution in autonomous mode. In other words, processor 11 does not determine or store the third position information and the third rotation angle information in manual mode. In this case, the third region refers to the region that is not the first region or the second region.

[0182] In the autonomous mode of each embodiment and its variations, when it is determined that the current camera area is contained within the third region (neither the first nor the second region), the processor 11 can also calculate the target rotation angle θt or ωt at the current position φ of the endoscope 2 in real time based on the current position φ, the first position information, the first rotation angle information, the second position information, and the second rotation angle information (SC13). Furthermore, when the current camera area is contained within any region of the first or second region (but not within the third region), the processor 11 does not calculate the target rotation angle θt or ωt in real time, but simply makes it consistent with the first or second rotation angle information. Therefore, the amount of position information and rotation angle information stored in the storage unit 13 during manual mode can be reduced, and only the third position information and third rotation angle information required for autonomous mode operation need to be calculated, thus reducing the load on the system.

[0183] Furthermore, if the current camera area is contained within the first or second area, the processor 11 can update the pre-stored first or second position information, or first or second rotation angle information, to the current position and rotation angle information. After updating, the endoscope 2 is moved, and if it is determined that the current camera area is contained within the first or second area, the updated first and second position information, first and second rotation angle information, can be used. When updating, the user only needs to issue an update instruction through the user interface 16. Therefore, even if a part of the patient's body is deformed due to pneumoperitoneum or positional adjustments, the position and rotation angle information can be updated to the correct information corresponding to the current situation.

[0184] In the above embodiments and variations, the processor 11 identifies specific tissues within the endoscopic image E and determines position information and rotation angle information based on the identified specific tissues. Alternatively, the position information and rotation angle information can be determined based on the position φ and rotation angle ω of the endoscope 2 at the moment of receiving each instruction.

[0185] That is, in manual mode, the surgeon positions the endoscope 2 at the desired location and rotation angle ω, and inputs a first instruction. The processor 11 determines the periphery of the endoscope 2's position φ at the moment the user interface 16 receives the first instruction as first position information, and determines the rotation angle ω of the endoscope 2 at the moment the user interface 16 receives the second instruction as first rotation angle information.

[0186] Similarly, the surgeon positions the endoscope 2 at other desired locations with a desired rotation angle ω and inputs a second instruction. The processor 11 determines the periphery of the endoscope 2's position φ at the moment the user interface 16 receives the second instruction as second position information, and determines the rotation angle ω of the endoscope 2 at the moment the user interface 16 receives the second instruction as second rotation angle information.

[0187] According to this structure, the surgeon can register any region within the examined body as the first region and the second region, and can further determine the position and rotation angle information based on the surgeon's intuition. Furthermore, even if the first and second regions do not contain specific tissue, the position and rotation angle information of the first and second regions can be arbitrarily determined and stored without performing the learning of Model 1b.

[0188] Alternatively, it can be determined using the positional information and rotation angle information of a specific tissue within the endoscopic image E, and the positional information and rotation angle information of the endoscope 2 at the time of the acceptance instruction.

[0189] For example, as described in the first to third embodiments and their variations, after determining the first position information and the second position information, as well as the first rotation angle information and the second rotation angle information based on specific tissues F and G in the endoscopic image E, the processor 11 may further determine the position information and rotation angle information of any region different from the first region and the second region according to the surgeon's instructions.

[0190] In the above embodiments and variations, the specific tissues are set as the aorta F and the pelvis G, but the specific tissues can be any organ or tissue with anatomical features, such as the uterus.

[0191] In the above embodiments and variations, position information and rotation angle information for two regions are stored. However, it is also possible to store position information and rotation angle information for three or more regions instead. This improves the accuracy when calculating position information and rotation angle information based on pre-stored information.

[0192] In the above embodiments and variations, the position φ of the endoscope 2, representing the location of the imaging area, is described using a two-dimensional polar coordinate system with the pivot point H as the origin. However, it can also be described using a three-dimensional polar coordinate system. That is, the endoscope 2 can be supported so that it can swing about a second pivot P2 that passes through the pivot point H and is perpendicular to the first pivot P1, and the position of the imaging area is described as (φ1, φ2). φ1 is the rotation angle about the first pivot P1, and φ2 is the rotation angle about the second pivot P2. In this case, the first position information, the second position information, and the third position information are three-dimensional information including the rotation angles φ1 and φ2, respectively.

[0193] In the above embodiments and variations, the position of the camera area can also be represented by other types of coordinate systems instead of polar coordinates. For example, the position of the camera area can also be described by a Cartesian coordinate system with aperture H as the origin.

[0194] In the above embodiments and variations, the coordinate system of the position φ of the imaging area is a global coordinate system fixed relative to the subject, but it can also be replaced by a relative coordinate system relative to the front end of the endoscope 2.

[0195] In the above embodiments and variations, the first position information and the second position information are determined and stored in the storage unit 13 in manual mode. However, it is also possible to store the first position information and the second position information in the storage unit 13 in advance before the operation.

[0196] Before surgery, sometimes examination images, such as CT images of the abdomen, are obtained that cover the area to be treated. A three-dimensional image of the abdominal cavity is generated by deconvolution of multiple CT images. Alternatively, based on such a preoperative three-dimensional image, first and second location information can be determined before surgery and stored in storage unit 13. In this case, steps SB4 and SB6 are omitted in manual mode.

[0197] Based on this structure, the computational load of processor 11 in manual mode can be reduced.

[0198] In the above embodiments and variations, the processor 11 may also save the endoscopic image E obtained by capturing the first region (i.e., the first endoscopic image) and the endoscopic image E obtained by capturing the second region (i.e., the second endoscopic image) in the storage unit 13 in manual mode. For example, in step SB3, the processor 11 saves one or more endoscopic images E that identify the aorta F as first endoscopic images in the storage unit 13. In addition, in step SB6, the processor 11 saves one or more endoscopic images E that identify the pelvis G as second endoscopic images in the storage unit 13.

[0199] In this case, the processor 11 can also determine, in autonomous mode, which of the first, second, and third regions the current imaging region is contained within, based on the first and second endoscopic images. That is, the processor 11 compares the current endoscopic image E with the first and second endoscopic images. Furthermore, the processor 11 determines that the current imaging region is contained within the first region if a first endoscopic image that is identical or similar to the current endoscopic image E exists, and determines that the current imaging region is contained within the second region if a second endoscopic image that is identical or similar to the current endoscopic image E exists.

[0200] In the above embodiments and variations, when the endoscopic image E contains a specific tissue, the processor 11 can also read information related to the rotation angle of the specific tissue from the database 1c stored in the storage unit 13, and rotate the endoscopic image E according to the read rotation angle information. The rotation angle is the angle about the center point of the endoscopic image E. According to this structure, the endoscopic image E can be rotated so that the specific tissue in the endoscopic image E is positioned at a predetermined rotation angle.

[0201] For example, database 1c records the types of one or more specific tissues other than the aorta F and the pelvis G, as well as the rotation angles of each type. Processor 11 identifies a specific tissue within the endoscopic image E, reads the rotation angle of the specific tissue from database 1c, and rotates the endoscopic image E to position the specific tissue at the rotation angle.

[0202] As an example, the uterus J, being a specific tissue, is preferably positioned above the endoscopic image E. Therefore, in the database 1c, a 90° angle corresponding to the 12 o'clock position is registered as the rotation angle of the uterus J. The processor 11 rotates the endoscopic image E so that the identified uterus J is positioned at a 90° position. Thus, when the uterus J is included in the endoscopic image E, the vertical direction of the endoscopic image E is automatically adjusted so that the uterus J is positioned at a 90° position.

[0203] In the above embodiments and variations, the rotation of the endoscope image E is controlled based on specific tissues F and G within the endoscope image E. However, in addition, the rotation of the endoscope image E can also be controlled based on the treatment device 6 within the endoscope image E.

[0204] For example, the processor 11 can operate in a first rotation mode that controls the rotation of the endoscopic image E according to specific tissues F and G, and in a second rotation mode that controls the rotation of the endoscopic image E according to the treatment device 6. Users such as surgeons can switch between the first and second rotation modes using the user interface 16.

[0205] In the second mode, the processor 11 detects the angle of the instrument 6 within the current endoscopic image E, and rotates the endoscopic image E through rotation of the endoscope 2 or image processing to make the angle of the instrument 6 equal to a predetermined target angle. The rotated endoscopic image E is then output to the display device 5 and displayed on the display screen 5a. The angle of the instrument 6 is, for example, the angle of the length axis of the instrument 6 relative to the horizontal line of the endoscopic image E.

[0206] In order for the surgeon to properly operate the instrument 6 while observing the endoscopic image E, it is important that the angle of the instrument 6 displayed in the endoscopic image E on the display screen 5a is appropriate. However, the angle of the instrument 6 in the endoscopic image E changes as the surgeon moves the instrument 6 or follows the posture changes of the endoscope 2 that controls the instrument 6.

[0207] The surgeon can switch from the first rotation mode to the second rotation mode as needed, thereby enabling the treatment instrument 6 in the endoscopic image E to be displayed on the display screen 5a at the target angle.

[0208] In the above-described embodiments and variations, the surgeon holds the treatment instrument 6 by hand and operates it manually, but this can be replaced by other methods, such as... Figure 14A and Figure 14BAs shown, the treatment device 6 is held and controlled by a second moving device 31, which is different from the moving device 3. In this case, the control device 1 can also obtain position information of the endoscope 2 and the treatment device 6 from the moving device 3 that moves the endoscope 2 and the second moving device 31 that moves the treatment device 6, respectively. Like the moving device 3, the second moving device 31 holds the treatment device 6 by means of a robotic arm or an electric holder, and changes the position and orientation of the treatment device 6 in three dimensions according to the control of the control device 101. The treatment device 6 can be as follows... Figure 14A As shown, it connects to the front end of the robotic arm and becomes an integral part of the robotic arm; it can also be like... Figure 14B As shown, it is separate from the robotic arm and is held by the robotic arm.

[0209] Explanation of reference numerals in the attached figures

[0210] 1 Control device

[0211] 11 processor

[0212] 12 memory

[0213] 13. Storage unit, recording medium

[0214] 14 input interfaces

[0215] 15 output interfaces

[0216] 16 User Interfaces

[0217] 1a Image Control Program

[0218] 1b Model completed learning

[0219] 1c database

[0220] 2. Endoscope

[0221] 2a camera element

[0222] 3 mobile devices

[0223] 3A robotic arm

[0224] Joints 3b and 3c

[0225] 3D angle sensor

[0226] 4 Endoscope Processors

[0227] 5 display devices

[0228] 5a display screen

[0229] 6. Treatment equipment

[0230] Positions A, B, D, and O

[0231] C-axis, visual axis

[0232] P1 First Pivot

[0233] P2 Second Pivot

[0234] E-endoscopic images

[0235] F major artery, first specific tissue

[0236] G pelvis, second specific tissue

[0237] H hole

Claims

1. An endoscope system comprising: An endoscope is inserted into a body to obtain endoscopic images of the body. A moving device that holds the endoscope and moves the endoscope; Storage department; and The control device has at least one processor. The storage unit stores first position information and first rotation angle information of a first region within the subject, and second position information and second rotation angle information of a second region within the subject that is different from the first region. The first rotation angle information specifies the rotation angle of the endoscopic image of the first region, and the second rotation angle information specifies the rotation angle of the endoscopic image of the second region. The at least one processor performs the following processing: Based on the first position information, the first rotation angle information, the second position information, the second rotation angle information, and the third position information of the third region within the subject, the third rotation angle information of the third region is calculated. The third region is a region different from the first and second regions. If the current imaging area captured by the endoscope is contained within the third region, the endoscope image is rotated according to the third rotation angle information. The rotated endoscopic image is output to a display device.

2. The endoscope system according to claim 1, wherein, The at least one processor rotates the endoscopic image through image processing.

3. The endoscope system according to claim 1, wherein, The moving device enables the endoscope to rotate about its optical axis. The at least one processor controls the moving device to rotate the endoscope about the optical axis, thereby rotating the endoscopic image.

4. The endoscope system according to claim 1, wherein, The at least one processor is capable of operating in a manual mode, allowing the user to manipulate the endoscope. In the manual mode, the at least one processor determines the first position information, the first rotation angle information, the second position information, and the second rotation angle information and stores them in the storage unit.

5. The endoscopic system according to claim 4, wherein, The at least one processor performs the following processing: The first position information and the first rotation angle information are determined based on a first specific tissue contained within the endoscopic image. The second position information and the second rotation angle information are determined based on a second specific tissue contained within the endoscopic image.

6. The endoscopic system according to claim 5, wherein, The storage unit stores a learned model obtained by machine learning, which is derived from the correspondence between images containing specific tissues and the types of those specific tissues. The at least one processor performs the following processing: The learned model stored in the storage unit is used to identify the first specific tissue and the second specific tissue within the endoscopic image. The first position information is determined based on the position of the imaging area of ​​the endoscopic image of the first specific tissue, and the first rotation angle information is determined based on the rotation angle of the first specific tissue within the endoscopic image. The second position information is determined based on the position of the imaging area of ​​the endoscopic image in which the second specific tissue is identified, and the second rotation angle information is determined based on the rotation angle of the second specific tissue within the endoscopic image.

7. The endoscope system according to claim 4, wherein, The control device also has a user interface that accepts user instructions. The at least one processor performs the following processing: The first position information is determined based on the position of the camera area when the user interface accepts the first instruction, and the first rotation angle information is determined based on the rotation angle of the endoscope around the optical axis when the user interface accepts the first instruction. The second position information is determined based on the position of the camera area when the user interface accepts the second instruction, and the second rotation angle information is determined based on the rotation angle of the endoscope around the optical axis when the user interface accepts the second instruction.

8. The endoscope system according to claim 4, wherein, The at least one processor stores the endoscopic image obtained by capturing the first region, i.e., the first endoscopic image, and the endoscopic image obtained by capturing the second region, i.e., the second endoscopic image, in the storage unit.

9. The endoscope system according to claim 8, wherein, The at least one processor determines, based on the first endoscope image and the second endoscope image stored in the storage unit, which of the first region, the second region, and the third region the current camera area is included in.

10. The endoscope system according to claim 1, wherein, The storage unit stores in advance the first location information and the second location information, which are determined based on examination images of the subject obtained before surgery.

11. The endoscope system according to claim 1, wherein, The at least one processor performs the following processing: When the current camera area is contained within the first area, the endoscopic image is rotated according to the first rotation angle information. When the current camera area is contained within the second area, the endoscope image is rotated according to the second rotation angle information.

12. The endoscopic system according to claim 11, wherein, The processor determines, based on the location of the camera area, the first location information, and the second location information, which of the first, second, and third regions the current camera area is included in.

13. The endoscope system according to claim 1, wherein, The endoscope is supported at a predetermined pivot point fixed relative to the subject, allowing it to swing about a first pivot. This swing of the endoscope about the first pivot allows the imaging area to move between a first area and a second area. The first position information, the second position information, and the third position information each include the rotation angle of the endoscope about the first pivot.

14. The endoscopic system according to claim 13, wherein, The endoscope is supported at a predetermined pivot point to allow it to swing about a second pivot perpendicular to the first pivot. The first position information, the second position information, and the third position information are all three-dimensional information, and also include the rotation angle of the endoscope around the second pivot.

15. The endoscopic system according to claim 13, wherein, The mobile device has one or more joints and one or more angle sensors for detecting the rotation angle of each of the one or more joints. The processor calculates the rotation angle of the endoscope around the first pivot based on the rotation angle detected by the one or more angle sensors.

16. The endoscopic system according to claim 1, wherein, The storage unit contains a database that maps specific tissue types to rotation angle information. If the specific tissue is contained within the endoscopic image of the third region, the processor performs the following processing: Read the rotation angle information corresponding to the type of the specific tissue in the endoscopic image from the database. The endoscopic image is rotated based on the read rotation angle information.

17. The endoscope system according to claim 1, wherein, The at least one processor performs the following processing: Calculate the positional relationship between the third position information and the first and second position information. The third rotation angle information is calculated based on the positional relationship, the first rotation angle information, and the second rotation angle information.

18. The endoscope system according to claim 3, wherein, When the rotation angle of the endoscope about the optical axis reaches the limit angle of the specified rotatable range, the at least one processor rotates the endoscope image through image processing.

19. The endoscope system according to claim 1, wherein, The endoscope is either a direct-viewing endoscope or an oblique-viewing endoscope.

20. The endoscope system according to claim 1, wherein, The endoscope has a bendable portion at its front end that can be electrically bent.

21. The endoscope system according to claim 1, wherein, When the current camera area is included in the first area, the at least one processor can update the first position information or the first rotation angle information to the current position information or rotation angle information of the camera area. When the current camera area is included in the second area, the at least one processor can update the second position information or the second rotation angle information to the current position information or rotation angle information of the camera area.

22. The endoscope system according to claim 4, wherein, In the manual mode, the at least one processor calculates the third position information and the third rotation angle information and stores them in the storage unit.

23. The endoscope system according to claim 1, wherein, The at least one processor is capable of operating in an autonomous mode, in which the endoscope is moved autonomously by controlling the moving device. The at least one processor calculates the third position information and the third rotation angle information in the autonomous mode.

24. A control device for controlling endoscopic images acquired by an endoscope and displayed on a display device, wherein, The control device has a storage unit and at least one processor. The storage unit stores first position information and first rotation angle information of a first region within the subject, and second position information and second rotation angle information of a second region within the subject that is different from the first region. The first rotation angle information specifies the rotation angle of the endoscopic image of the first region, and the second rotation angle information specifies the rotation angle of the endoscopic image of the second region. The at least one processor performs the following processing: The third rotation angle information of the third region is calculated based on the first position information, the first rotation angle information, the second position information, the second rotation angle information, and the third position information of the third region within the subject body. The third region is a region different from the first region and the second region. If the current imaging area captured by the endoscope is contained within the third region, the endoscope image is rotated according to the third rotation angle information. The rotated endoscopic image is output to a display device.

25. A computer-readable non-transitory recording medium storing a control program for causing a computer to execute a control method, said control method using first position information and first rotation angle information of a first region within a subject, and second position information and second rotation angle information of a second region within the subject that is different from the first region, to control an endoscopic image acquired by an endoscope and displayed on a display device, wherein, The first rotation angle information specifies the rotation angle of the endoscopic image in the first region, and the second rotation angle information specifies the rotation angle of the endoscopic image in the second region. The control method includes the following steps: The third rotation angle information of the third region is calculated based on the first position information, the first rotation angle information, the second position information, the second rotation angle information, and the third position information of the third region within the subject body. The third region is a region different from the first region and the second region. If the current imaging area captured by the endoscope is contained within the third region, the endoscope image is rotated according to the third rotation angle information; and The rotated endoscopic image is output to a display device.

Images