Surgical system and control method of a surgical system

Abstract

The surgical system (100) includes: a camera (1) for photographing the object to be treated; a display (4) for displaying a screen (4a) of the object to be treated photographed by the camera (1); and a control (3) for controlling the display range of the object to be treated displayed on the screen (4a). The control (3) calculates the position of the cut end of the object to be treated based on the image of the object to be treated obtained by the camera (1) and moves the display range of the object to a position in which the cut end is positioned within the screen (4a).

Description

Technical Field

[0001] This invention relates to surgical systems and methods for controlling surgical systems. Background Technology

[0002] Previously, surgical systems that use robots to move endoscopes were known (for example, see Patent Document 1). When anatomical features such as blood vessels of the heart are covered by fat, or when anatomical features are located outside the endoscopic field of view, they cannot be seen in the endoscopic image. Patent Document 1 discloses a technique that determines the endoscope's orientation based on anatomical features in a preoperative image, controls a robot to position the endoscope in the determined orientation, thereby making the anatomical features visible.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent No. 6629186 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Due to factors such as gravity and the softness of the object being treated, the object may deform during treatment, causing changes in its appearance as displayed in the endoscopic image. For example, in hepatectomy, the position of the cutting line and the cut end on the screen changes as the liver is dissected. Similarly, in S-shaped colectomy, the position of the dissection line on the screen changes as the fascia is dissected. Since the surgeon operates the instruments based on the endoscopic images, it can be difficult to perform the treatment when the object's appearance changes.

[0008] The present invention was made in view of the above circumstances, and its object is to provide a surgical system and a control method for the surgical system that can continuously display the object being treated in a state suitable for treatment on the screen regardless of deformation of the object being treated.

[0009] means for solving problems

[0010] To achieve the above objectives, the present invention provides the following technical solutions.

[0011] One aspect of the present invention is a surgical system comprising: a camera device for capturing images of a subject; a display device for displaying an image of the subject captured by the camera device; and a control device for controlling the display range of the subject displayed on the image, the control device calculating the position of a cut-off end of the subject based on the image of the subject acquired by the camera device, and moving the display range of the subject to a position in which the cut-off end is positioned within the image.

[0012] According to this method, a camera device inserted into the patient's body captures an image of the object to be treated, and the image is displayed on a screen. While observing the object displayed on the screen, the surgeon cuts the object away from the patient using a treatment instrument inserted into the patient's body. As the cutting proceeds, the object deforms, and thus, the position of the cut end relative to the camera device may sometimes change.

[0013] During the cutting of the object to be processed, the control device calculates the position of the cutting end based on the image. If the position of the cutting end deviates from a predetermined reference position within the screen, the control device moves the display area of ​​the object to be processed to a position that positions the cutting end within the predetermined reference position within the screen. That is, the display area moves following the changing position of the cutting end, and the cutting end remains positioned at the predetermined reference position within the screen. As a result, the processing position can be continuously displayed on the screen in a state suitable for processing, regardless of deformation of the object to be processed during cutting.

[0014] In the above scheme, the surgical system includes a moving device for moving the camera device, and the control device calculates the amount of change of the position of the cut end relative to the predetermined reference position, and controls the moving device to move the camera device to a position where the amount of change is zero.

[0015] According to this structure, when the position of the cut-off end changes due to deformation of the object being processed, the moving device moves the camera device, thereby moving the field of view of the camera device, which is the display range of the object being processed, to a position where the cut-off end is positioned within the frame. In this way, by automatically following the changing position of the cut-off end with the field of view of the camera device, the cut-off end can be continuously displayed at the specified reference position within the frame.

[0016] In the above method, the control device may pre-calculate the position of the cut-off end within the screen before calculating the position of the cut-off end of the object to be disposed of and before moving the display range, and set the pre-calculated position of the cut-off end as the predetermined reference position.

[0017] According to this structure, the position of the cutting end on the screen, which is pre-calculated before the position changes due to deformation of the object being treated, is set as a predetermined reference position. Therefore, for example, the predetermined reference position can be set so that the cutting end is positioned in a position that makes it easy for the surgeon to cut away from the object being treated.

[0018] In the above scheme, the control device sets the specified reference position to any position within a region of ±15% longitudinally and / or ±15% laterally from the center of the screen.

[0019] Based on this structure, the cut-off end can be continuously displayed in the central area of ​​the image.

[0020] In the above method, the control device may also calculate the orientation of the mark on the object to be disposed of in the screen, and rotate the display range of the object to be disposed of to an orientation in which the mark is configured as a predetermined reference orientation in the screen.

[0021] Sometimes, a mark indicating a predetermined cutting line is marked on the object to be disposed of. The orientation of the mark, like the position of the cutting end, sometimes changes as the cutting progresses. According to the above structure, during the cutting of the object to be disposed of, the orientation of the mark within the screen is calculated by a control device. If the orientation of the mark deviates from a predetermined reference orientation of the screen, the display range is rotated by the control device to the orientation that positions the mark within the predetermined reference orientation of the screen. That is, the display range rotates following the changing orientation of the mark, and the mark remains continuously positioned within the screen at the predetermined reference orientation. Therefore, the disposal position can be continuously displayed on the screen in a state more suitable for disposal, regardless of deformation of the object to be disposed of during cutting.

[0022] In the above method, the control device may pre-calculate the orientation of the mark in the screen before calculating the orientation of the mark, and set the pre-calculated orientation of the mark as the specified reference orientation.

[0023] According to this structure, the orientation of the marker on the screen, which is pre-calculated before the orientation of the marker changes due to deformation of the object being treated, is set as a predetermined reference orientation. Therefore, for example, it is possible to set the predetermined reference orientation so that the marker is configured to be easily cut away from the object being treated by the surgeon.

[0024] In the above scheme, the control device may set the specified reference orientation as follows: the orientation is such that the end of the mark that is farthest from the cut-off end is positioned within a region of ±15% longitudinally or ±15% laterally from the center of the image.

[0025] Based on this structure, markers can be displayed continuously, roughly parallel to the vertical or horizontal orientation of the image.

[0026] In the above-described manner, the control device may calculate the position of the cutting end as the location of the intersection point between the mark marked on the outer surface of the object being treated and the edge of the cutting surface of the object. Alternatively, in the above-described manner, the control device may calculate the position of the end of the mark marked on the outer surface of the object being treated within the frame that is closer to the center of gravity of the cutting surface of the object being treated as the location of the cutting end. Alternatively, in the above-described manner, the control device may calculate the position of the intersection point between the edges of the cutting surfaces on both sides of the object being treated as the location of the cutting end.

[0027] Based on these structures, the position of the cut-off end can be calculated through simple image processing and computation.

[0028] In the above-described manner, the control device may simultaneously maintain the size of the display range of the object being processed while moving the display range of the object being processed.

[0029] For example, when the display area of ​​the object being handled is moved by moving the camera device, the size of the display area changes when the distance between the camera device and the object changes. According to the above structure, this undesirable situation can be eliminated, and a fixed-size display area of ​​the object being handled can be continuously displayed on the screen.

[0030] Alternatively, in the above-described manner, the control device may move the display range of the object being processed by processing the image.

[0031] According to this structure, when it is impossible to move the camera device to the target position using a moving device, the position of the cut-off end can be continuously displayed at a predetermined reference position on the screen through image processing. Alternatively, the display range can be moved without moving the camera device, but only through image processing.

[0032] In the above technical solution, if it is impossible to move the display range of the object being processed to a position that places the cut-off end within the screen at a predetermined reference position, the control device updates the predetermined reference position.

[0033] According to this structure, when the range of motion of the camera device is limited, it is sometimes impossible to move the cut-off end to a predetermined reference position by moving the camera device and image processing. In such cases, by updating the predetermined reference position, that is, setting another position within the frame as the new predetermined reference position, the cut-off end can be continuously displayed at the new predetermined reference position within the frame.

[0034] Another aspect of the present invention is a control method for a surgical system, the control method being executed by a control device of the surgical system, wherein the surgical system includes: a camera device for capturing images of a subject to be treated; and a display device having an image of the subject to be treated captured by the camera device, the control method of the surgical system calculating the position of the cut-off end of the subject to be treated based on the image of the subject to be treated acquired by the camera device, and moving the display area of ​​the subject to be treated to a position in which the cut-off end is positioned within the image.

[0035] Invention Effects

[0036] According to the present invention, the following effect is achieved: the object to be disposed of can be continuously displayed on the screen in a state suitable for disposal, regardless of the deformation of the object to be disposed of. Attached Figure Description

[0037] Figure 1 This is an external view of the overall structure of a surgical system according to one embodiment of the present invention.

[0038] Figure 2 It is shown Figure 1 A block diagram of the structure of the control device for the surgical system.

[0039] Figure 3A It is a diagram representing the initial state of the object being disposed of.

[0040] Figure 3B It is a diagram that shows the state of deformation of the object being processed as the cutting proceeds.

[0041] Figure 3C It is a diagram showing the state in which the endoscope is moved relative to the cut end and the markings to the appropriate position and orientation.

[0042] Figure 4A yes Figure 3A Endoscopic images of the object being treated.

[0043] Figure 4B yes Figure 3B Endoscopic images of the object being treated.

[0044] Figure 4C yes Figure 3C Endoscopic images of the object being treated.

[0045] Figure 5A yes Figure 1 A flowchart of the control method for the surgical system.

[0046] Figure 5B yes Figure 5A The flowchart for the characteristic quantity calculation routine.

[0047] Figure 6 This is a diagram illustrating the calculation method for the cut-off end.

[0048] Figure 7 yes Figure 5A A flowchart of a variation of the control method.

[0049] Figure 8 yes Figure 5A A flowchart of a variation of the control method.

[0050] Figure 9 This is another example illustrating the reference position.

[0051] Figure 10A This is a diagram illustrating other calculation methods for the cut-off end.

[0052] Figure 10B This is a diagram illustrating other calculation methods for the cut-off end. Detailed Implementation

[0053] Hereinafter, a surgical system and a method for controlling the surgical system according to one embodiment of the present invention will be described with reference to the accompanying drawings.

[0054] [Structure of Surgical System 100]

[0055] The surgical system 100 of this embodiment is a system for assisting in the handling of the endoscope 1 and the instruments 5 during laparoscopic surgery. The surgical system 100 has an autonomous control function for the endoscope 1, and in particular, an autonomous movement function that optimizes the field of view for each surgical scenario by moving the endoscope 1.

[0056] like Figure 1 As shown, the surgical system 100 includes: an endoscope 1 serving as a camera device, which captures images of the object to be treated A inside the body of the patient P; a moving device 2, which moves the endoscope 1; a control device 3, which controls the endoscope 1 and the moving device 2; and a display device 4, which displays the image of the object to be treated A captured by the endoscope 1. Furthermore, the camera device is not limited to the endoscope 1, but may also be any device that captures images of the object to be treated A inside the body of the patient P.

[0057] Endoscope 1 is a rigid endoscope, and its front end is equipped with a lens and imaging element for photographing the organ or tissue being treated, A. Endoscope 1 is connected to control device 3 via a signal line passing through the inside of the moving device 2. Endoscope 1 receives control signals from control device 3 for controlling endoscope 1 and sends endoscopic image data to control device 3.

[0058] The mobile device 2 is an electrically driven robotic arm with at least one bending joint 2a, and the base end of the endoscope 1 is connected to the front end of the robotic arm 2.

[0059] Display device 4 is a known display device such as a liquid crystal display, and has a screen 4a. Display device 4 is connected to control device 3, and displays the endoscopic image of the object A to be treated, input from control device 3, on screen 4a. Display device 4 may also be a head-mounted display or a projector.

[0060] By bending the joint 2a of the robotic arm 2, the endoscope 1 moves, thereby shifting the field of view of the endoscope 1, i.e., the display area of ​​the object A being treated, shown on the screen 4a. The movement of the robotic arm 2 is controlled by the control device 3.

[0061] like Figure 2 As shown, the control device 3 includes at least one processor 3a, a memory 3b, a storage unit 3c, an input interface 3d, an output interface 3e, and a network interface 3f, similar to a central processing unit.

[0062] Endoscopic images transmitted from endoscope 1 are sequentially input to control device 3 via input interface 3d and sequentially output to display device 4 via output interface 3e. Thus, endoscopic images of the object A being treated, captured by endoscope 1, are displayed on screen 4a.

[0063] Storage unit 3c is a non-volatile recording medium such as ROM (read-only memory) or hard disk, storing programs and data required for processor 3a to perform processing. The functions of control device 3, described later, are implemented by reading the program into memory 3b and executing it by processor 3a. Some functions of control device 3 can also be implemented using dedicated logic circuits, etc.

[0064] like Figure 3A and Figure 3B As shown, in surgeries such as liver resections where object A is dissected, due to the influence of gravity G and the softness of object A, at least a portion of object A deforms or shifts during the dissection. The result is as follows: Figure 4A and Figure 4B As shown, sometimes the position, orientation, and size of the display area of ​​the object A within screen 4a gradually change. Figures 3A to 3C This indicates a liver resection, specifically the removal of the parenchyma from the liver of the patient (subject A). Figures 3A to 3C In the equation, gravity G is the downward force on the paper.

[0065] The control device 3 calculates a first characteristic quantity, a second characteristic quantity, and a third characteristic quantity. The first characteristic quantity is a quantity representing the composition of the object A within the screen 4a. The second characteristic quantity is a quantity representing the orientation of the object A within the screen 4a. The third characteristic quantity is a quantity representing the size of the display area of ​​the object A displayed in the screen 4a, i.e., the size of the field of view of the endoscope 1.

[0066] Specifically, the control device 3 calculates the position of the cutting end B within the screen 4a as a first feature quantity. The cutting end B is the leading end of a cutting line formed by the treatment instrument 5. The position of the cutting end B is the coordinate of the cutting end B in the screen coordinate system Σs fixed to the screen 4a. Furthermore, the control device 3 calculates the orientation of the mark C within the screen 4a (the rotation angle of the mark C along the plane of the screen 4a) as a second feature quantity. The mark C is a predetermined cutting line marked on the outer surface of the treatment object A before cutting it off; it is, for example, a dashed or solid line. The mark C is marked, for example, by cauterizing the outer surface of the treatment object A using an electrosurgical scalpel. Moreover, the control device 3 calculates the distance between the tip of the endoscope 1 and the cutting end B as a third feature quantity.

[0067] The specific calculation methods for each characteristic quantity will be described in detail later.

[0068] The control device 3 acquires one endoscope image from the sequentially input endoscope images and calculates the aforementioned first, second, and third feature values ​​based on the acquired endoscope image. Then, the control device 3 controls the robotic arm 2 based on the calculated first, second, and third feature values, thereby moving the field of view of the endoscope 1 so that each feature value is aligned with a predetermined reference. As a result, the object being treated, A, is continuously displayed on screen 4a according to the predetermined reference.

[0069] Furthermore, before calculating the characteristic quantities and moving the field of view of the endoscope 1, the control device 3 sets a predetermined reference based on the endoscopic image D. The method for setting the predetermined reference will be described in detail later.

[0070] [Control method of Surgical System 100]

[0071] Next, the control method of the surgical system 100 executed by the control device 3 will be described.

[0072] like Figure 5A As shown, the control method of the surgical system 100 includes: reference setting steps S1 to S3, setting references for each of the three feature quantities; feature quantity calculation steps S4 and S5, calculating the three feature quantities based on the endoscopic image D; and field of view adjustment step S6, adjusting the field of view of the endoscope 1 by controlling the moving device 2.

[0073] The baseline setting steps include step S1 of acquiring the endoscopic image D, step S2 of calculating three feature quantities, and step S3 of setting the baseline for each feature quantity.

[0074] In step S1, the control device 3 retrieves one endoscope image D from the sequentially input endoscope images D. For example... Figure 4AAs shown, the endoscopic image D acquired at this time is an image prior to the movement of the severing end B and the marker C due to the deformation of the object being treated A. For example, image acquisition is performed in response to the operator's input to the control device 3.

[0075] Next, in step S2, three feature quantities are calculated based on the acquired endoscopic image D. Specifically, as follows: Figure 5B As shown, step S2 includes: step S21 of identifying the mark C and the edge F of the cutting surface E in the endoscopic image D; step S22 of calculating the first feature quantity; step S23 of calculating the second feature quantity; and steps S24 to S26 of calculating the third feature quantity.

[0076] In step S21, as Figure 6 As shown, the control device 3 identifies the mark C and the edge F of the dissecting surface E by performing image processing on the endoscopic image D. The mark C and the dissecting surface E have a different color from the outer surface and surrounding tissue of the object being treated A; therefore, the control device 3 can identify the mark C and the dissecting surface E based on color. At this time, it is difficult to distinguish the two dissecting surfaces E from each other through image processing; the two dissecting surfaces E are identified as a single surface. Therefore, the edge F is identified as a single line.

[0077] Next, in step S22, the control device 3 calculates the position of the intersection point of the mark C and the edge F within the screen 4a as the position of the cut-off end B. The position of the cut-off end B is represented by coordinates in the screen coordinate system Σs.

[0078] like Figure 6 As shown, when C is marked as a dashed line, a solid line connecting the points can also be generated, and the position of the intersection of the solid line and the edge F can be calculated.

[0079] Furthermore, when a closed loop edge F or multiple edges F are identified, the positions of multiple intersection points can be calculated. In this case, the position of the intersection point closest to marker C can also be used as the position of the cutting end B.

[0080] Next, in step S23, the control device 3 detects the positions of any two different points on the mark C within the screen 4a. The positions of the two points are represented as coordinates in the screen coordinate system Σs. Then, the control device 3 calculates the vector connecting the two points as the orientation of the mark C within the screen 4a.

[0081] The order of steps S22 and S23 is arbitrary, and step S22 can be performed after step S23.

[0082] Next, in step S24, the control device 3 detects the position of the tip of the endoscope 1 in the base coordinate system Σr of the robot holding the endoscope 1. The base coordinate system Σr is a coordinate system fixed relative to the non-movable part of the robotic arm 2 (e.g., the base end of the robotic arm 2). For example, the control device 3 obtains the rotation angle of each joint 2a by angle sensors installed on each joint 2a of the robotic arm 2, and detects the position of the tip of the endoscope 1 based on the rotation angle. The position of the tip of the endoscope 1 is represented by coordinates in the robot's base coordinate system Σr.

[0083] Next, in step S25, the control device 3 transforms the position of the detached end B calculated in step S22 into its position in the robot's base coordinate system Σr. Additionally, the control device 3 transforms the orientation of the marker C calculated in step S23 into its orientation in the robot's base coordinate system Σr.

[0084] Next, in step S26, the control device 3 calculates the distance between the position of the tip of the endoscope 1 and the position of the cut-off end B obtained in steps S24 and S25, and stores the calculated distance in the storage unit 3c. Additionally, the control device 3 stores the position of the cut-off end B after the change in step S25 and the orientation of the mark C in the storage unit 3c.

[0085] Next, in step S3, the control device 3 sets the position of the cut-off end B stored in the storage unit 3c as a predetermined reference position, serving as a reference for the first feature quantity. Furthermore, the control device 3 sets the orientation of the mark C stored in the storage unit 3c as a predetermined reference orientation, serving as a reference for the second feature quantity. Additionally, the control device 3 sets the distance stored in the storage unit 3c as a predetermined reference distance, serving as a reference for the third feature quantity.

[0086] Next, the feature calculation step is performed. The feature calculation step includes step S4 of re-acquiring the endoscopic image D and step S5 of re-calculating the three feature quantities based on the acquired endoscopic image D.

[0087] In step S4, the control device 3 acquires the endoscopic image D again. The acquired endoscopic image D at this time is as follows: Figure 4B The image shown is an endoscope image taken from the endoscope image D in step S1 after some time has passed; it is the endoscope image at the time of dissection.

[0088] Next, in step S5, the control device 3 performs the same processing as steps S21 to S26 based on the re-acquired endoscopic image D, thereby calculating three feature quantities.

[0089] Next, in the field of view adjustment step S6, the control device 3 compares the three feature quantities calculated in step S5 with the reference, and calculates the change of each feature quantity relative to the reference. Then, the control device 3 calculates the target position and target orientation of the endoscope 1 that makes the change of each feature quantity zero and equal to the reference. That is, the target position and target orientation are the following positions and orientations of the endoscope 1: the cut-off end B is positioned in the reference position within the screen 4a, the marker C is positioned in the reference orientation within the screen 4a, and the distance from the cut-off end B to the tip of the endoscope 1 is equal to the reference distance.

[0090] Next, the control device 3 calculates the amount and direction of movement of the endoscope 1 to move it to the target position and target posture, and calculates the amount of motion of the robotic arm 2 (specifically, the rotation of each joint 2a) to achieve the calculated amount and direction of movement. Then, the control device 3 generates a control signal for the calculated amount of motion of the robotic arm 2 and sends the control signal to the robotic arm 2.

[0091] Therefore, as Figure 3C As shown, robotic arm 2 moves in response to control signals. Consequently, endoscope 1 moves within the abdominal cavity to a target position and pose where changes in the three characteristic quantities are canceled out, and the field of view of endoscope 1 shifts. The result is as follows: Figure 4C As shown, the display area of ​​the object to be treated (A) displayed on screen 4a is moved and rotated, the cutting end (B) is positioned at a predetermined reference position, and the marker (C) is positioned at a predetermined reference orientation. Furthermore, the distance from the tip of the endoscope 1 to the cutting end (B) is adjusted to a predetermined reference distance, and the size of the display area of ​​the object to be treated (A) displayed on screen 4a is maintained at a predetermined size.

[0092] Thus, according to this embodiment, as the dissection proceeds and the treated object A, such as the liver, deforms and its characteristic dimensions change, the endoscope 1 automatically moves to a position and direction that counteracts the change in characteristic dimensions. That is, the field of view of the endoscope 1 automatically moves in tandem with the dissection tip B and the marker C. Therefore, during the dissection of the treated object A along the marker C, the position of the dissection tip B, the direction of the marker C, and the size of the field of view are maintained at their respective references. For example, the position of the dissection tip B is continuously displayed in the center of the screen 4a, and the marker C is continuously displayed in the vertical direction of the screen 4a.

[0093] During the dissection of object A, the surgeon controls the position and direction of movement of the instrument 5 based on the endoscopic image D of object A displayed on screen 4a. According to this embodiment, object A can be continuously displayed on screen 4a in a state that is easy for the surgeon to handle, regardless of deformation of object A during dissection.

[0094] When the range of motion of endoscope 1 is limited, it is sometimes impossible to move endoscope 1 to a target position and target posture where the three characteristic quantities are equal to the reference, respectively. For example, if the surgical system 100 is equipped with a function to prevent interference between endoscope 1 and surrounding objects (such as surrounding tissues or other instruments), it is impossible to move endoscope 1 to a position where interference between endoscope 1 and surrounding objects is predicted to occur. In this case, control device 3 can generate an image in which the change in each of the three characteristic quantities relative to the reference is zero by processing the endoscope image D, and output the generated image to display device 4 for display on screen 4a. Thus, as when moving endoscope 1, the display range of the object being treated A displayed on screen 4a can be moved and rotated.

[0095] Specifically, such as Figure 7 As shown, steps S7 to S10 can also be added after step S5.

[0096] In step S7, the control device 3 determines whether the endoscope 1 can be moved to the target position and target posture. If it is determined that the endoscope 1 cannot be moved to the target position and target posture (No in step S7), the control device 3 determines in step S8 whether an endoscope image D with zero change in feature quantity can be generated through image processing. If an endoscope image D with zero change in feature quantity can be generated through image processing (Yes in step S8), the control device 3 performs image processing in step S9. Specifically, the control device 3 generates an image with zero change in feature quantity by cropping, scaling, and rotating the endoscope image D acquired in step S4.

[0097] If it is impossible to generate an image where the change in feature quantity is zero through image processing (No in step S8), the control device 3 may also update the reference of the feature quantity in step S10. For example, the control device 3 may also set the feature quantity calculated in step S5 as the new reference.

[0098] In the above embodiment, feature quantities are pre-calculated based on the endoscopic image D before steps S4 to S6, and the pre-calculated feature quantities are set as a reference. However, this can be replaced by other methods, such as... Figure 8 As shown, a baseline is preset. Figure 9 This represents an example of a pre-defined first characteristic quantity and a reference for a second characteristic quantity.

[0099] The reference position, serving as the reference for the first characteristic quantity, is set at a predetermined point or a predetermined portion within the frame 4a (step S11). The reference position can also be the center point or central region of the frame 4a. For example... Figure 9As shown, the central region is preferably within ±15% of the horizontal dimension of the image 4a from the center point, and within ±15% of the vertical dimension of the image 4a. The central region is not limited to a rectangle; for example, it can also be a circular or elliptical region centered on the center point of the image 4a.

[0100] The reference orientation, serving as the reference for the second characteristic quantity, is set as follows: the end of the mark C within the screen 4a that is farther from the end B is positioned within a range of ±15% laterally from the center point of the screen 4a (within the range I to I) (step S12). Thus, the mark C is maintained in a direction parallel or approximately parallel to the longitudinal direction of the screen 4a. To maintain the mark C in a direction parallel or approximately parallel to the lateral direction of the screen 4a, the reference orientation can also be set such that the more distant end of the mark C is positioned within a range of ±15% longitudinally from the center point of the screen 4a (within the range II to II).

[0101] The reference distance, which serves as the reference for the third characteristic quantity, is set using the same method as steps S21, S22, S24 to S26, and S3 described above (step S13).

[0102] In the above embodiment, the intersection of the mark C and the edge F of the cutting surface E is used as the position of the cutting end B. However, the method for calculating the position of the cutting end B is not limited to this and other methods can also be used.

[0103] Figure 10A and Figure 10B Another example illustrating the calculation method for the cut-off end B.

[0104] exist Figure 10A In the process, the control device 3 calculates the position of the centroid H of the cutting surface E, and calculates the position of the end of the two ends of the mark C in the screen 4a that is closer to the centroid H as the position of the cutting end B.

[0105] exist Figure 10B In this process, the control device 3 calculates the position of the intersection point of the edge F of the cutting surface E on one side and the edge F of the cutting surface E on the other side as the position of the cutting end B. Alternatively, the control device 3 can approximate the edge F of the cutting surface E on each side with a straight line and calculate the position of the intersection point of the two straight lines as the position of the cutting end B.

[0106] The upper edge F of the cutting surface E is an upwardly convex curve, and the cutting end B is located at the vertex of edge F. Therefore, other methods capable of detecting the vertex of edge F can also be used. For example, the control device 3 can also calculate the maximum point of the curve representing edge F as the position of the cutting end B.

[0107] In the above embodiments, an example of directly marking the object being treated with the mark C was described. However, the mark C can also be an imaginary mark that is overlaid on the endoscopic image D. That is, the mark C can also be a mark displayed on screen 4a by image processing of the endoscopic image D. The mark C is formed using known techniques based on the information of the object being treated and displayed on screen 4a overlaid on the endoscopic image D.

[0108] In the above embodiment, a laparoscopic surgery in which the liver parenchyma is removed is used as an example for the treatment object A. However, the treatment object A and the surgery using the surgical system 100 are not limited to this. The surgical system 100 can also be used in other treatment objects and surgeries.

[0109] For example, in an S-shaped colectomy, the dissection line can be moved within the abdominal cavity when the fascia is dissected from the S-shaped colon. In this case, the field of view of endoscope 1 can be followed by the dissection line according to the progress of the dissection, allowing endoscope 1 to move autonomously so that the dissection line is positioned, oriented, and at an easily manageable location.

[0110] In addition, the surgical system 100 may include endoscopes of types other than laparoscopy and can be used for endoscopic surgeries other than laparoscopic surgeries.

[0111] In the above embodiment, the display range of the object to be processed A displayed on the screen 4a is moved and rotated by moving the endoscope 1. However, it is also possible to move and rotate the display range only by image processing.

[0112] For example, endoscope 1 acquires a large-scale endoscopic image D, a portion of which is displayed on screen 4a. In this case, by moving and rotating the portion of endoscopic image D displayed on screen 4a, the same effect as when endoscope 1 is moved can be achieved.

[0113] Marker description

[0114] 100 Surgical System

[0115] 1. Endoscope (camera device)

[0116] 2. Mobile devices, robotic arms

[0117] 3. Control device

[0118] 4 Display devices

[0119] 4a screen

[0120] 5. Handling equipment

[0121] A. Target of Disposal

[0122] B. Cut-off end

[0123] C mark

[0124] D Endoscopic image

[0125] G gravity

Claims

1. A surgical system, wherein, The surgical system has the following features: The camera device, and the object it captures; A display device having the ability to display an image of the object being processed captured by the camera device; and A control device that controls the display range of the object being processed displayed on the screen. The control device calculates the position of the cut-off end of the object to be processed based on the image of the object acquired by the camera device, and moves the display area of ​​the object to a position that positions the cut-off end within the frame. The control device calculates the orientation of the mark on the object being processed within the screen, and rotates the display area of ​​the object being processed to an orientation that positions the mark within the screen as a predetermined reference orientation. The calculation of the position of the cut-off end of the object to be disposed of includes: The control device calculates the position of the intersection point of the mark and the edge of the cutting surface of the object to be disposed of as the position of the cutting end, or... The control device calculates the position of the end of the mark within the image that is closer to the centroid of the cutting surface of the object being disposed of, as the position of the cutting end, or... The control device calculates the position of the intersection point between the edges of the cutting surfaces on both sides of the object to be treated as the position of the cutting end.

2. The surgical system according to claim 1, wherein, The surgical system includes a moving device for moving the camera device. The control device calculates the change in the position of the cut-off end relative to the specified reference position, and controls the moving device to move the camera device to a position where the change is zero.

3. The surgical system according to claim 1, wherein, Before calculating the position of the cut-off end of the object to be disposed of and before moving the display range, the control device pre-calculates the position of the cut-off end within the screen and sets the pre-calculated position of the cut-off end as the specified reference position.

4. The surgical system according to claim 1, wherein, The control device sets the specified reference position to any position within a region of ±15% longitudinally and / or ±15% laterally, starting from the center of the image.

5. The surgical system according to claim 1, wherein, Before calculating the orientation of the marker, the control device pre-calculates the orientation of the marker within the screen and sets the pre-calculated orientation of the marker as the specified reference orientation.

6. The surgical system according to claim 1, wherein, The control device sets the specified reference orientation as follows: the orientation causes the end of the mark that is farthest from the cut-off end to be positioned within a region of ±15% longitudinally or ±15% laterally from the center of the image.

7. The surgical system according to claim 1, wherein, The control device maintains the size of the display area of ​​the object being processed while moving the display area of ​​the object being processed.

8. The surgical system according to claim 1, wherein, The control device moves and rotates the display area of ​​the object being processed by processing the image.

9. A control method for a surgical system, wherein the control method is executed by a control device of the surgical system, wherein, The surgical system includes: a camera device for capturing images of the object being treated; and a display device for displaying images of the object being treated captured by the camera device. In the control method of the surgical system, Based on the image of the object being processed acquired by the camera device, the position of the cut-off end of the object being processed is calculated, and the display area of ​​the object being processed is moved to a position that positions the cut-off end within the frame as a predetermined reference position. Calculate the orientation of the mark on the object being processed within the screen, and rotate the display area of ​​the object being processed to an orientation that positions the mark within the screen as a predetermined reference orientation. The calculation of the position of the cut-off end of the object to be disposed of includes: The position of the intersection point between the mark and the edge of the cutting surface of the object being disposed of is calculated as the position of the cutting end, or... The position of the cutting end is calculated as the position of the end of the mark within the image that is closer to the centroid of the cutting surface of the object being disposed of. The position of the intersection point between the edges of the cutting surfaces on both sides of the object to be treated is calculated as the position of the cutting end.

10. The control method for the surgical system according to claim 9, wherein, The surgical system includes a moving device for moving the camera device. Moving the display range of the object to be processed includes the following steps: calculating the target position of the camera device at a predetermined reference position within the screen where the cut-off end is positioned; and moving the camera device to the target position by controlling the moving device.

11. The control method for the surgical system according to claim 9, wherein, The control method of the surgical system further includes the following steps before calculating the position of the cut-off end of the object to be treated and moving the display range: pre-calculating the position of the cut-off end within the screen and setting the pre-calculated position of the cut-off end as the specified reference position.

12. The control method for the surgical system according to claim 9, wherein, The control method of the surgical system further includes the following steps: setting the specified reference position as any position within a region of ±15% longitudinally and / or ±15% laterally from the center of the image.

13. The control method for the surgical system according to claim 9, wherein, The control method of the surgical system further includes the following steps before calculating the orientation of the mark and rotating the display range: pre-calculating the orientation of the mark within the screen and setting the pre-calculated orientation of the mark as the specified reference orientation.

14. The control method for the surgical system according to claim 9, wherein, The specified reference orientation is set as follows: the orientation is such that the end of the mark that is farthest from the cut-off end is positioned within a region of ±15% longitudinally or ±15% laterally from the center of the image.

15. The control method for the surgical system according to claim 9, wherein, While maintaining the size of the display area of ​​the object being processed, the display area of ​​the object being processed is moved.

16. The control method for the surgical system according to claim 9, wherein, If it is not possible to move the display area of ​​the object being processed to a position that allows the cut-off end to be positioned within the specified reference position on the screen, the specified reference position is updated.

17. The control method for the surgical system according to claim 9, wherein, The image processing causes the display area of ​​the object to be processed to move and rotate.

Images