Force sensation display device, force sensation display method, and program

Abstract

A haptic display device that easily realizes reduction of damage to tissue is provided with an image acquisition unit (51) that acquires photographic information of a treatment site; a haptic acquisition unit (52) that acquires haptic information including information on the magnitude and direction of force acting on at least one surgical instrument (21R, 21L) that treats the treatment site; a generation unit (53) that generates a haptic display based on the haptic information, the haptic display including a force display that represents the magnitude of the force and a direction display that represents the direction of the force; a detection unit (54) that detects a surgical instrument region in a display unit (12) in which the surgical instrument is displayed based on the photographic information; and a superimposition unit (53) that generates display information that is information that superimposes the haptic display on the surgical instrument region and displays it on the display unit.

Description

Technical Field

[0001] This disclosure relates to force display devices, force display methods, and procedures. Background Technology

[0002] Endoscopic surgery and laparoscopic surgery using surgical instruments such as endoscopes, laparoscopes, or forceps are becoming increasingly common. In the following text, surgical instruments such as endoscopes, laparoscopes, or forceps will be referred to as "surgical instruments." Furthermore, endoscopic surgery or laparoscopic surgery using surgical instruments will be referred to as "endoscopic surgery, etc."

[0003] In procedures such as endoscopic surgery, surgeons use handheld surgical instruments to perform the operation. In addition, they may use medical robots for surgical assistance, such as those disclosed in Patent Document 1. Surgery using medical robots for surgical assistance is also known as robot-assisted surgery.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2001-104333 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] As is well known, in procedures such as endoscopic surgery, it is difficult for surgeons to control the force exerted by surgical instruments on organs and tissues. For example, it is well known that surgeons have difficulty controlling the force exerted by surgical instruments on tissues, such as the force used to push or grasp them. Therefore, it is possible to apply force to tissues beyond the intended force, thereby causing damage. In the following text, information related to the force exerted by surgical instruments on organs and tissues will also be referred to as "force information".

[0009] As a method for the practitioner to obtain the aforementioned force information, the following methods are known: in addition to using an image display device that displays images taken by an endoscope, a force display device specifically designed to display force information to the practitioner is also used. In this method, the practitioner needs to move their gaze from the image display device to the force display device in order to obtain the force information. Therefore, the following problem exists: unintentional damage to tissues may be caused when moving the gaze.

[0010] Furthermore, methods for overlaying force information onto an image display are known. In these methods, if force information is overlaid onto an organized portion of the image display, it becomes difficult to visually identify the tissue obscured by the overlaid force information. Therefore, there is a potential problem of causing unintentional damage to tissue that is difficult to visually identify.

[0011] Patent Document 1 disclosed a technique for overlaying force information onto a portion of an instrument, which is equivalent to a surgical instrument, displayed on an image display. Specifically, it disclosed a technique for overlaying an indicator showing the magnitude of the force applied to tissue, as well as information indicating the joint angle or rotation angle of the instrument. The information displayed in this case has the following problem: due to insufficient information, it is difficult to predict the damage to the tissue.

[0012] This disclosure discloses examples of a force-sensing display device, force-sensing display method, and procedure that can easily reduce damage to tissues.

[0013] Problem Solving Methods

[0014] One force-sensing display device disclosed herein includes: an image acquisition unit that acquires imaging information output from an imaging unit that captures images of a affected area; a force-sensing acquisition unit that acquires force-sensing information, the force-sensing information including information on the magnitude and direction of a force exerted on at least one surgical instrument treating the affected area; a generation unit that generates a force-sensing display based on the force-sensing information, the force-sensing display being a display format displayed on a display unit, and including a force display indicating the magnitude of the force and a direction display indicating the direction of the force; a detection unit that detects a surgical instrument area based on the imaging information, the surgical instrument area being an area in an image displayed on the display unit where the surgical instrument is displayed; and an overlay unit that generates display information, the display information being information that overlays the force-sensing display onto the surgical instrument area and displays it on the display unit.

[0015] One aspect of this disclosure is a force-sensing display method for a force-sensing display device having an image acquisition unit, a force-sensing acquisition unit, a generation unit, a detection unit, and an overlay unit. The method includes: an image acquisition step, wherein the image acquisition unit acquires imaging information output from an imaging unit that captures an image of a affected area; a force-sensing acquisition step, wherein the force-sensing acquisition unit acquires force-sensing information, the force-sensing information including information on the magnitude and direction of a force acting on at least one surgical instrument treating the affected area; a generation step, wherein the generation unit generates a force-sensing display based on the force-sensing information, the force-sensing display being a display form displayed on a display unit, and including a force display indicating the magnitude of the force and a direction display indicating the direction of the force; a detection step, wherein the detection unit detects a surgical instrument area based on the imaging information, the surgical instrument area being an area in an image displayed on the display unit where the surgical instrument is displayed; and an overlay step, wherein the overlay unit generates display information, the display information being information that overlays the force-sensing display onto the surgical instrument area and displays it on the display unit.

[0016] One aspect of this disclosure provides a program for enabling a computer to perform the following functions: an image acquisition function for acquiring imaging information output from an imaging unit that captures images of a affected area; a force acquisition function for acquiring force information, the force information including information on the magnitude and direction of a force exerted on at least one surgical instrument treating the affected area; a generation function for generating a force display based on the force information, the force display being a display form shown on a display unit and including a force display indicating the magnitude of the force and a direction display indicating the direction of the force; a detection function for detecting a surgical instrument area based on the imaging information, the surgical instrument area being an area in the image displayed on the display unit where the surgical instrument is displayed; and a superposition function for generating display information, the display information being information that superimposes the force display onto the surgical instrument area and displays it on the display unit.

[0017] The program can be stored in ROM or RAM that can be installed into a computer and loaded from that ROM or RAM for use. The program can also be loaded into a computer and used via a network. Furthermore, the program can be recorded on any form of recording medium that can be read by a computer for use. Recording media include, for example, portable semiconductor memory (e.g., USB memory).

[0018] According to the force display device, force display method, and procedure described above, the force display, which includes a force display indicating the magnitude of the force and a direction display indicating the direction of the force, is displayed in a manner that is superimposed on the image displayed on the display unit, showing the area of ​​the surgical instrument. Therefore, it is easy to grasp the magnitude and direction of the force applied by the surgical instrument to the affected area. In other words, it is easy to grasp the damage caused to the tissue of the affected area by the surgical instrument.

[0019] Furthermore, the force sensor display is superimposed on the surgical instrument area, making it less likely for the affected area displayed on the display to be obscured by the force sensor display. Therefore, unintentional contact between the surgical instruments and the affected area is less likely to occur, thus reducing the risk of injury due to tissue contact.

[0020] In one aspect of this disclosure, the generating unit preferably generates the force display, which preferably includes a force display that uses at least one of color type, color brightness, and translucency to represent the magnitude of the force.

[0021] The force display device constructed as described above uses one of the following—color type, color brightness, and translucency—or a combination thereof, to represent the magnitude of force, thereby making it easy to intuitively grasp the magnitude of force.

[0022] In one aspect of this disclosure, the generating unit preferably generates the force display, which preferably includes a force display that uses numerical values ​​to represent the magnitude of the force.

[0023] The force display device constructed as described above uses numerical values ​​to represent the magnitude of force, thereby accurately representing the magnitude of force.

[0024] In one aspect of this disclosure, the generating unit preferably generates the force display, which preferably includes a force display that uses the size of the displayed numerical value to represent the magnitude of the force.

[0025] The force display device constructed as described above uses numerical values ​​to represent the magnitude of force. Therefore, compared with the case where only numerical values ​​are used to represent the magnitude of force, it can not only accurately represent the magnitude of force, but also make it easier to grasp the magnitude of force more intuitively.

[0026] In one embodiment of this disclosure, a storage unit is preferably provided, which preferably stores at least one of the colors and patterns of the surgical instruments, and the detection unit preferably detects the surgical instrument area in the image displayed on the display unit based on at least one of the colors and patterns of the surgical instruments stored in the storage unit.

[0027] The force-sensing display device described above detects the surgical instrument area based on at least one of the color and style of the surgical instrument, thereby facilitating the detection of the surgical instrument area. Furthermore, it easily improves the detection accuracy of the surgical instrument area.

[0028] In one embodiment of this disclosure, the force information preferably includes information on the magnitude and direction of a first force acting on a rod-shaped shaft portion of the surgical instrument. The generation unit preferably generates a first force display, which preferably includes a first force display indicating the magnitude of the first force and a first direction display indicating the direction of the first force. The detection unit preferably detects the area in the image displayed on the display unit where the shaft portion is displayed as a first surgical instrument area. The overlay unit preferably generates the display information, which is information that overlays the first force display onto the first surgical instrument area and displays it on the display unit.

[0029] The force display device configured as described above displays a first force display related to the shaft of the surgical instrument in a manner that is superimposed on the area of ​​the first surgical instrument displaying the shaft. This makes it easy to grasp the magnitude and direction of the force applied by the shaft to the affected area.

[0030] In one embodiment of this disclosure, the force information preferably includes information on the magnitude and direction of a second force applied to an end effector portion, the end effector portion being disposed at the end of the shaft portion and used to treat the affected area; the generation unit preferably generates a second force display, the second force display including a second force display indicating the magnitude of the second force and a second direction display indicating the direction of the second force; the detection unit preferably detects the area in the image displayed on the display unit where the end effector portion is displayed as a second surgical instrument area; and the overlay unit preferably generates the display information, which is information that overlays the second force display onto the second surgical instrument area and displays it on the display unit.

[0031] The force display device configured as described above displays a second force display associated with the end effector of the surgical instrument in a manner that is superimposed on the area of ​​the second surgical instrument displaying the end effector. This makes it easy to grasp the magnitude and direction of the force applied by the end effector to the affected area.

[0032] The effects of the invention

[0033] According to the force display device, force display method and procedure disclosed herein, it is easy to grasp the magnitude and direction of the force applied by surgical instruments to the affected area, and it is easy to achieve the effect of reducing damage to tissues. Attached Figure Description

[0034] Figure 1 This is a block diagram illustrating the configuration of the medical robot according to the first embodiment of this disclosure.

[0035] Figure 2 This is a diagram illustrating the structure of the instruments displayed on an endoscope monitor.

[0036] Figure 3 This is a block diagram illustrating the structure of a force-sensing display device.

[0037] Figure 4 This is a diagram illustrating the first and second force sensory displays shown on the endoscope monitor.

[0038] Figure 5 This is a diagram illustrating other display examples in the first direction.

[0039] Figure 6 This is a diagram illustrating other display examples in the first direction.

[0040] Figure 7 This is a diagram illustrating other display examples in the first direction.

[0041] Figure 8 This is a diagram illustrating other display examples in the first direction.

[0042] Figure 9 This is a flowchart illustrating the computational processing of the force sensory display.

[0043] Figure 10 This diagram illustrates the configuration of a force-sensing display device used in laparoscopic surgery.

[0044] Figure 11 This is a block diagram illustrating the configuration of a force display device according to a modified example of the present disclosure.

[0045] Explanation of reference numerals in the attached figures

[0046] 12…Endoscopic monitor (display unit); 21R, 21L, 121R, 121L…Instruments (surgical instruments);

[0047] 22R, 22L…shaft section; 23R, 23L…end actuator section; 35…shooting section;

[0048] 50, 50A…force sensor display device; 51…image acquisition unit; 52…force sensor acquisition unit;

[0049] 53… Generation section; 54, 54A… Detection section; 55… Overlay section; 57A… Storage section;

[0050] 41R, 41L… First strength display (strength display);

[0051] 42R, 42L… First direction display (direction display);

[0052] 46R, 46L… Second strength display (strength display);

[0053] 27R, 27L… Surgical Instrument Area 1 (Surgical Instrument Area)

[0054] 28R, 28L… Second Surgical Instrument Area (Surgical Instrument Area)

[0055] S101…Image acquisition step; S102…Detection step; S103…Force sensing acquisition step;

[0056] S104…Generation step; S105…Overlay step Detailed Implementation

[0057] The following embodiments illustrate examples of implementations within the scope of this disclosure. The scope of this disclosure is not limited to the specific configurations or structures shown in the following embodiments.

[0058] The arrows and diagonal lines used to indicate directions in the accompanying drawings are symbols used to facilitate understanding of the relationships between the drawings and the shapes of the components or parts. Therefore, the technical scope of this disclosure is not limited to the directions indicated in the drawings. Drawings marked with diagonal lines are not necessarily sectional views.

[0059] For each component or part that is marked with a symbol, at least one is provided unless otherwise stated as "one". That is, if not otherwise stated, two or more of the component may be provided. The force display device and medical robot disclosed herein include constituent elements such as components or parts marked with symbols and illustrated structural parts.

[0060] [First Embodiment]

[0061] Reference Figures 1-9 The force display device 50 of the first embodiment of this disclosure and the medical robot 1 having the force display device 50 will be described. The medical robot 1 of this embodiment is a robot that assists in surgery, such as endoscopic surgery.

[0062] In this embodiment, the force display device 50 is described as being integrated with the medical robot 1. The force display device 50 may also have a separate structure that allows it to be installed and removed from the medical robot 1.

[0063] like Figure 1 As shown, the medical robot 1 is equipped with a master device 10, a slave device 20, and a force display device 50.

[0064] The master device 10 is a surgeon's console for operating the slave device 20, which is used by the surgeon performing surgery using the medical robot 1. The master device 10 is connected to the slave device 20 so that signals related to operation can be communicated.

[0065] In this embodiment, the example is described where the main device 10 is installed in a different room than the room where the subordinate device 20 is installed. However, the main device 10 may also be installed in the same room as the room where the subordinate device 20 is installed.

[0066] The main side device 10 is equipped with at least an electrosurgical knife pedal 11 and an endoscope monitor 12. In addition, the main side device 10 is provided with a part for the surgeon to operate, namely, an operating part (not shown) used when performing operations such as moving or holding the two instruments 21R and 21L described later.

[0067] The electrosurgical pedal 11 is an instrument for the surgeon to operate, and also controls the operation of the electrosurgical generator 25 described later. In this embodiment, the example will be described where the electrosurgical pedal 11 is positioned under the surgeon's feet and is used by the surgeon to control the operation of the electrosurgical generator 25 by stepping on it. The electrosurgical pedal 11 may also have other configurations besides being operated by the surgeon by stepping on it.

[0068] The endoscope monitor 12 is a display unit used by the surgeon during operation and is an instrument that displays images captured by the endoscope 31, described later. The endoscope monitor 12 can be any commonly used display device; however, its display format is not limited. The endoscope monitor 12 can be mounted on the main device 10 and integrated with it, or it can be mounted on a device other than the main device 10.

[0069] The slave device 20 is a bedside robotic arm system installed in the operating room and used to perform surgery according to the surgeon's instructions input via the master device 10. The slave device 20 includes mechanisms for mounting the instruments 21R, 21L, and endoscope 31 (described later), for moving the instruments 21R, 21L, and endoscope 31, and for driving the instruments 21R, 21L, and endoscope 31.

[0070] In this embodiment, the example described is a subordinate device 20 equipped with a mechanism that moves instruments 21R, 21L, and the endoscope 31 using air pressure, or a mechanism that drives instruments 21R, 21L, and the endoscope 31 using air pressure. Other mechanisms, such as a mechanism that moves instruments 21R, 21L, and the endoscope 31 using an electric motor, or a mechanism that drives instruments 21R, 21L, and the endoscope 31 using an electric motor, may also be provided in the subordinate device 20.

[0071] The slave device 20 may be equipped with two instruments 21R and 21L, and an endoscope 31. In this embodiment, the configuration in which the slave device 20 is equipped with two instruments 21R and 21L, and an endoscope 31 will be described as an example.

[0072] The number of instruments that can be mounted on the slave device 20 can be one or more. In addition, the number of endoscopes that can be mounted on the slave device 20 can be two or more.

[0073] Instruments 21R and 21L are instruments mounted on the auxiliary device 20 and are surgical instruments used to perform various procedures during surgery according to the surgeon's instructions. For example... Figure 2 As shown, the device 21R is provided with at least a shaft portion 22R and an end effector portion 23R. The device 21L is provided with at least a shaft portion 22L and an end effector portion 23L.

[0074] Both shaft portions 22R and 22L are rod-shaped components. An end effector portion 23R and an end effector portion 23L are disposed at the ends of shaft portions 22R and 22L, respectively. In this embodiment, the configuration where the ends of shaft portions 22R and 22L containing the end effector portions 23R and 23L are inserted into the patient's abdominal cavity will be described as an example. Shaft portions 22R and 22L may have the same diameter and length, or they may have different diameters and lengths.

[0075] In this embodiment, the configuration of the end effector 23R as an electrosurgical scalpel for cutting or performing other procedures on the affected area, and the end effector 23L as forceps for holding the affected area or separating adhered lesions, will be described as an example. Furthermore, the configuration of the end effector 23R connected to receive power supplied from the electrosurgical scalpel generator 25 will also be described as an example. The end effectors 23R and 23L may also have other configurations besides an electrosurgical scalpel or forceps for treating the affected area.

[0076] like Figure 1 As shown, the electrosurgical scalpel generator 25 is an instrument that is operably connected to the electrosurgical scalpel pedal 11 and supplies power to the end effector unit 23R. The power supplied from the electrosurgical scalpel generator 25 is also used for hemostasis at the lesion site that has been cut by the end effector unit 23R. The electrosurgical scalpel generator 25 can adopt a configuration commonly used as an instrument for supplying power to the end effector unit of the electrosurgical scalpel.

[0077] The endoscope 31 is an instrument mounted on the subordinate device 20 and is inserted into the patient's body cavity to take pictures of the affected area. The endoscope 31 is provided with a imaging section 35 for taking pictures.

[0078] The imaging unit 35 is equipped with an imaging element, a light-emitting element, and a computing element. The imaging element converts the image of the affected area guided by the endoscope 31 into an electrical signal and outputs the electrical signal as imaging information. The light-emitting element emits light that shines onto the affected area. The computing element is used to control the imaging element and the light-emitting element.

[0079] As the imaging element, the element commonly used in imaging elements such as solid-state imaging elements (e.g., CCD: Charge Coupled Device; photocoupler) can be used. As the light-emitting element, the element commonly used in light-emitting elements such as light-emitting diodes (LEDs) can be used.

[0080] In this embodiment, the example is described with the imaging element and the light-emitting element located in the imaging section 35, which is different from the endoscope 31. However, it is also possible to have at least one of the imaging element and the light-emitting element located in the endoscope 31.

[0081] like Figure 1 As shown, the force display device 50 is disposed between the slave device 20 and the endoscope monitor 12, and between the imaging unit 35 and the endoscope monitor 12. Furthermore, the force display device 50 is a device that generates display information, which is information that superimposes the force display onto the imaging information output from the imaging unit 35. The force display shows the magnitude and direction of the force acting on the information instruments 21R and 21L.

[0082] The force-sensing display device 50 is an information processing device such as a processor, microcomputer, personal computer, or server, which has a CPU (central processing unit), ROM, RAM, input / output interfaces, etc.

[0083] The program stored in the aforementioned storage device such as ROM enables the CPU, ROM, RAM, and input / output interface to work together, and at least serves as... Figure 3 The image acquisition unit 51, force acquisition unit 52, generation unit 53, detection unit 54, overlay unit 55, and output unit 56 shown in the diagram perform their functions.

[0084] The image acquisition unit 51 is an interface for acquiring shooting information output from the imaging unit 35. The image acquisition unit 51 is connected to the imaging unit 35 in a manner that enables information communication. Furthermore, the image acquisition unit 51 is connected to the detection unit 54 and the overlay unit 55 in a manner that allows it to output the acquired shooting information to the detection unit 54 and the overlay unit 55.

[0085] The force acquisition unit 52 is an interface for acquiring force information, which includes the magnitude and direction of the force acting on the instruments 21R and 21L. The force acquisition unit 52 is connected to the shaft sensor 61R, the end effector sensor 62R, the shaft sensor 61L, and the end effector sensor 62L in a manner that enables information communication. The force acquisition unit 52 is connected to the generation unit 53 in a manner that allows it to output the acquired force information to the generation unit 53. In the following description, the force acting on the shaft of the instrument is referred to as the first force, and the force acting on the end effector is referred to as the second force.

[0086] The shaft sensor 61R is a sensor that measures force information including the magnitude and direction of a first force acting on the shaft portion 22R of the instrument 21R. The end effector sensor 62R is a sensor that measures force information including the magnitude of a second force acting on the end effector portion 23R of the instrument 21R.

[0087] The shaft sensor 61L is a sensor that measures force information including the magnitude and direction of a first force acting on the shaft portion 22L of the instrument 21L. The end effector sensor 62L is a sensor that measures force information including the magnitude of a second force acting on the end effector portion 23L of the instrument 21L.

[0088] In this embodiment, the configuration is described as follows: the shaft sensor 61R, the end effector sensor 62R, the shaft sensor 61L, and the end effector sensor 62L are sensors that measure the air pressure in the mechanism for moving the instruments 21R, 21L, and endoscope 31 provided in the slave device 20, and the air pressure in the mechanism for driving the instruments 21R, 21L, and endoscope 31 provided in the slave device 20.

[0089] The shaft sensor 61R, end effector sensor 62R, shaft sensor 61L, and end effector sensor 62L can also be other sensors used to measure the magnitude and direction of force, such as pressure sensors.

[0090] The generation unit 53 is a calculation unit that generates a force display based on the force information acquired by the force acquisition unit 52. The force display includes both force and direction displays. In this embodiment, the example of the generation unit 53 generating a first force display and a second force display will be described (see reference). Figure 4The first force display includes a first force display 41R, 41L indicating the magnitude of the first force and a first direction display 42R, 42L indicating the direction of the first force; the second force display includes a second force display 46R, 46L indicating the magnitude of the second force. The generation unit 53 is connected to the overlay unit 55 in such a way that the generated first force display and second force display can be output to the overlay unit 55.

[0091] The first force displays 41R and 41L are displayed on the endoscope monitor 12 and indicate the magnitude of the first force. The first direction displays 42R and 42L are displayed on the endoscope monitor 12 and indicate the direction of the first force.

[0092] The second force displays 46R and 46L are displayed on the endoscope monitor 12 and are a way of indicating the magnitude of the second force. Details of the first force displays 41R and 41L, the first direction displays 42R and 42L, and the second force displays 46R and 46L will be explained later.

[0093] The detection unit 54 is a processing unit that detects the surgical instrument area based on the acquired imaging information. The surgical instrument area is the area where instruments 21R and 21L are displayed when the imaging information is displayed on the endoscope monitor 12.

[0094] like Figure 2 As shown, the surgical instrument area includes the area of ​​display shaft portions 22R and 22L, namely the first surgical instrument area 27R and 27L; and the area of ​​display end effector portions 23R and 23L, namely the second surgical instrument area 28R and 28L. The detection unit 54 is connected to the overlay unit 55 in a manner that enables it to output information related to the detected first surgical instrument area 27R and 27L and the second surgical instrument area 28R and 28L to the overlay unit 55.

[0095] In this embodiment, the case where the detection unit 54 detects the first surgical instrument regions 27R and 27L and the second surgical instrument regions 28R and 28L by performing image analysis processing on the captured information will be described as an example. As for the image analysis processing, the processing method commonly used when detecting a specified region based on the captured information can be adopted.

[0096] The overlay unit 55 is a processing unit that generates display information that overlays the generated first force sensor display and second force sensor display onto the detected first surgical instrument areas 27R, 27L and second surgical instrument areas 28R, 28L. The overlay unit 55 is connected to the output unit 56 such that it can output the generated display information to the output unit 56. The display information is image information displayed on the endoscope monitor 12. The process of overlaying the force sensor display onto the surgical instrument areas in the overlay unit 55 can employ commonly used overlay processing.

[0097] like Figure 3 As shown, the output unit 56 is an interface for outputting the generated display information to the endoscope monitor 12. The output unit 56 is connected to the endoscope monitor 12 in a manner that enables information communication. The output unit 56 can be configured in a manner commonly found in interfaces.

[0098] Next, the method for displaying force sensation in the force sensation display device 50 of the medical robot 1, which includes the force sensation display device 50 with the above-described configuration, will be described. Here, for ease of explanation, the process of displaying force sensation when a force is applied to the shaft portion 22R and the end effector portion 23R of the instrument 21R will be described. Furthermore, the same process is performed when a force is applied to the shaft portion 22L and the end effector portion 23L of the instrument 21L.

[0099] As explained below Figure 9 The processing involves the CPU, ROM, RAM, and input / output interface of the force display device 50 coordinating actions to be performed by a program stored in the aforementioned storage device such as ROM.

[0100] If the process of displaying force via force display device 50 is initiated, then as follows Figure 9 As shown, the image acquisition unit 51 performs image information acquisition processing (S101: image acquisition step). In this embodiment, firstly, as... Figure 1 As shown, the image of the affected area guided by the endoscope 31 is incident on the imaging unit 35.

[0101] The imaging unit 35 converts the incident image into imaging information as an electrical signal and outputs the converted imaging information. For example... Figure 3 As shown, the image acquisition unit 51 acquires the output shooting information. The image acquisition unit 51 outputs the acquired shooting information to the detection unit 54 and the overlay unit 55.

[0102] like Figure 9 As shown, if image information is acquired, the detection unit 54 performs processing to detect the surgical instrument area (S102: detection step). In this embodiment, the detection unit 54 performs detection based on the acquired image information, such as... Figure 2The calculation and processing of the first surgical instrument area 27R and the second surgical instrument area 28R shown.

[0103] The computational processing for detecting the first surgical instrument region 27R and the second surgical instrument region 28R can employ detection processing typically used when detecting specific regions based on image information. For example... Figure 3 As shown, the first surgical instrument region 27R and the second surgical instrument region 28R that have been detected are output from the detection unit 54 to the superposition unit 55.

[0104] On the other hand, such as Figure 9 As shown, the force acquisition unit 52 performs a force acquisition process (S103: force acquisition step) to acquire force information. In this embodiment, as... Figure 2 as well as Figure 3 As shown, if a force is applied to the shaft portion 22R of the device 21R, the shaft sensor 61R measures the magnitude and direction of the first force acting on the shaft portion 22R. The shaft sensor 61R outputs force information, which includes information on the magnitude and direction of the first force.

[0105] If a force is applied to the end effector section 23R of the device 21R, the end effector sensor 62R measures the magnitude of the second force applied to the end effector section 23R. The end effector sensor 62R outputs force information, which includes information about the magnitude of the second force.

[0106] The force acquisition unit 52 acquires force information output from the shaft sensor 61R and the end effector sensor 62R. The acquired force information is then output from the force acquisition unit 52 to the generation unit 53.

[0107] If force information is obtained, then as follows Figure 9 As shown, the generation unit 53 performs calculations to generate a first force sensory display and a second force sensory display (S104: generation step). In this embodiment, the generation unit 53 performs calculations to generate a first force sensory display and a second force sensory display based on the input force sensory information. Figure 4 The calculation and processing of the first force display 41R, the first direction display 42R, and the second force display 46R shown. For example... Figure 3 As shown, the first force display 41R, the first direction display 42R, and the second force display 46R generated are output from the generation unit 53 to the superposition unit 55.

[0108] In this embodiment, the first force display 41R will be described as having two display modes: displaying at least a portion of the first surgical instrument area 27R with a color that changes according to the magnitude of the force; and displaying a numerical value indicating the magnitude of the force with the size of the value changing according to the magnitude of the force. More specifically, the example will be a case where the area whose color changes is an area with the same shape as the first surgical instrument area 27R. Here, the size of the numerical value refers to the size of the text representing the magnitude of the force.

[0109] Furthermore, taking the second force display 46R as an example, it will be explained in two ways: displaying at least a portion of the second surgical instrument area 28R, with the color changing according to the magnitude of the force; and displaying a numerical value indicating the magnitude of the force, with the size of the value changing according to the magnitude of the force. More specifically, it will be explained in the case where the area whose color changes is an area with the same shape as the second surgical instrument area 28R.

[0110] In this embodiment, the method of changing color according to the magnitude of force is described as an example where the color changes from green to yellow as the force increases, and then from yellow to red. However, the order in which colors are changed and the selection of the displayed color are not limited to the examples in this embodiment.

[0111] The method of changing the size of a numerical value according to the magnitude of the force is illustrated by an example of a method that increases the size of the displayed numerical value in a proportional manner. However, the method of changing the size is not limited to the example in this embodiment.

[0112] The display of at least a portion of the area of ​​the first surgical instrument region 27R in the first force display 41R, where the color changes according to the magnitude of the force, and the display of at least a portion of the area of ​​the second surgical instrument region 28R in the second force display 46R, where the color changes according to the magnitude of the force, can be a display of areas where the brightness of the color changes according to the magnitude of the force. For example, it can be a display method that changes from dark to light according to the magnitude of the force.

[0113] Furthermore, the display of at least a portion of the first surgical instrument region 27R in the first force display 41R, where the color changes according to the magnitude of the force, and the display of at least a portion of the second surgical instrument region 28R in the second force display 46R, where the color changes according to the magnitude of the force, can be a display of a region whose translucency changes according to the magnitude of the force. For example, it can be a display method that changes from transparent to translucent and then to opaque according to the magnitude of the force.

[0114] In this embodiment, the example will be described where the first direction display 42R is an arrow indicating the direction of force. The first direction display 42R can also be any other display method commonly used when indicating direction, besides an arrow.

[0115] For example, such as Figure 5 As shown, the first direction display 42R, 42L can be a display method that shows the direction of the force using a triangle. This triangle is an equilateral triangle with two equal sides longer than the length of one base. The direction of the vertex angle between the two equal sides indicates the direction of the force.

[0116] like Figure 6 As shown, the first direction display 42R, 42L can also be a combination of three L-shaped structures indicating the direction of force. In other words, it can be a display of small, medium, and large L-shaped structures. The number of L-shaped structures can be one, two, or four.

[0117] In this assembly, L-shaped shapes of different sizes are arranged in order of size. The direction pointed to by the corners of the L-shaped shapes indicates the direction of the force. Furthermore, when there are more than two L-shaped shapes, the direction of the force is indicated by the arrangement of the smaller L-shaped shapes towards the larger L-shaped shapes.

[0118] The display state of the three L-shaped aggregates can remain fixed regardless of the passage of time, or it can change over time. Specifically, it can repeatedly change over time from displaying only small L-shapes, then displaying small and medium L-shapes, then displaying small, medium, and large L-shapes, and then returning to displaying only small L-shapes.

[0119] like Figure 7 As shown, the first direction display 42R, 42L can also be a display method of four arc-shaped aggregates indicating the direction of force. The number of arcs can be one, two, three, or even more than five.

[0120] In this assembly, arcs of different sizes are arranged in order of size. The direction pointed to by the bisecting line of the central angle of an arc indicates the direction of the force. Furthermore, when there are more than two arcs, the direction of the force is also indicated by the arrangement of the smaller arcs towards the larger arcs.

[0121] The display state of the four arc-shaped aggregates can remain fixed regardless of the passage of time, or it can change over time, similar to the L-shaped aggregate. Specifically, it can repeatedly change from displaying only small arcs to displaying successively larger arcs over time, and then return to displaying only small arcs.

[0122] like Figure 8 As shown, the first direction, 42R and 42L, can also be displayed as a collection of three circles indicating the direction of force. The number of circles can be two or more. Within this collection, circles of different sizes are arranged in order of size. The direction of the force is represented by the arrangement of the smaller circles towards the larger ones.

[0123] The display state of the three circular aggregates can remain fixed regardless of the passage of time, or it can change over time, similar to the L-shaped aggregate. Specifically, it can repeatedly change from displaying only small circles to displaying successively larger circles, and then return to displaying only small circles over time.

[0124] If the first surgical instrument area 27R and the second surgical instrument area 28R are detected and the first force sensory display and the second force sensory display are generated, then as follows Figure 9 As shown, the overlay unit 55 performs calculations to generate display information (S105: overlay step). In this embodiment, as... Figure 4 As shown, the overlay unit 55 performs calculations to generate display information that superimposes the first force display 41R and the first direction display 42R onto the first surgical instrument area 27R. Furthermore, the overlay unit 55 also performs calculations to generate display information that superimposes the second force display 46R onto the second surgical instrument area 28R.

[0125] The first force display 41R and the first direction display 42R are preferably displayed in a manner that accommodates them within the first surgical instrument area 27R. The second force display 46R is preferably displayed in a manner that accommodates them within the second surgical instrument area 28R.

[0126] The first force display 41R and the first direction display 42R can also be displayed in a manner that extends beyond the first surgical instrument area 27R. The second force display 46R can also be displayed in a manner that extends beyond the second surgical instrument area 28R.

[0127] If display information is generated, then as follows: Figure 9 As shown, the output unit 56 performs the process of outputting display information to the endoscope monitor 12 (S106). The endoscope monitor 12 displays information based on the input display information, such as... Figure 4 The image shown.

[0128] If a display message is output, then such as Figure 9As shown, the force display device 50 performs a process (S107) to determine whether the input for ending the force display has been entered. If no input for ending the force display has been entered (NO), the force display device 50 returns to S101 and repeats the above process. If the input for ending the force display has been entered (YES), the force display device 50 ends the process of displaying force.

[0129] According to the force display device 50 described above, a first force display 41R and a first direction display 42R are superimposed on the first surgical instrument area 27R, and a second force display 46R is superimposed on the second surgical instrument area 28R. A first force display 41L and a first direction display 42L are superimposed on the first surgical instrument area 27L, and a second force display 46L is superimposed on the second surgical instrument area 28L.

[0130] Therefore, it is easy to control the magnitude and direction of the force applied to the affected area by instruments 21R and 21L. In other words, it is easy to control the damage to the tissues caused by instruments 21R and 21L. This makes it easier to reduce the damage to the tissues.

[0131] Furthermore, the first force display 41R, the first direction display 42R, and the second force display 46R are superimposed on the first surgical instrument area 27R and the second surgical instrument area 28R. The first force display 41L, the first direction display 42L, and the second force display 46L are superimposed on the first surgical instrument area 27L and the second surgical instrument area 28L. Therefore, the affected area displayed on the endoscopic monitor 12 is less likely to be obstructed by the first force display and the second force display. Unintentional contact between the instruments 21R, 21L and the affected area is less likely to occur, thus reducing the risk of injury due to tissue contact.

[0132] The first force display 41R and the first direction display 42R are superimposed on the first surgical instrument area 27R, and the first force display 41L and the first direction display 42L are superimposed on the first surgical instrument area 27L. Therefore, it is easy to grasp the magnitude and direction of the force applied to the affected area by the shafts 22R and 22L.

[0133] The second force display 46R is superimposed on the second surgical instrument area 28R, and the second force display 46L is superimposed on the second surgical instrument area 28L. Therefore, it is easy to grasp the magnitude of the force applied to the affected area by the end effector parts 23R and 23L.

[0134] The first force display 41R and the second force display 46R use different colors to represent the magnitude of the force, making it easy to intuitively grasp the magnitude of the force.

[0135] The first force display 41R and the second force display 46R are used to represent the magnitude of force, thus accurately representing the magnitude of the force.

[0136] The first force display 41R and the second force display 46R use numerical values ​​to represent the magnitude of force. Therefore, compared with the case where the magnitude of force is only represented by numerical values, the magnitude of force can not only be represented accurately, but also be more easily and intuitively grasped.

[0137] In the above embodiment, the configuration of the force display device 50 superimposing a first force display including first force displays 41R, 41L and first direction displays 42R, 42L, and a second force display including second force displays 46R, 46L on the endoscope monitor 12 is described as an example. However, it is also possible to superimpose only one of the first force display and the second force display on the endoscope monitor 12.

[0138] In addition to the second force displays 46R and 46L, the second force display may also include a second direction display indicating the direction of the second force. In this case, the second force display including the second force displays 46R and 46L and the second direction display is superimposed on the endoscope monitor 12.

[0139] In the above embodiments, an example of applying the force-sensing display device 50 to surgery using the medical robot 1 has been described. However, as... Figure 10 As shown, the force display device 50 can also be used in laparoscopic surgery.

[0140] When used in laparoscopic surgery, unlike the instruments 21R and 21L operated by the subordinate device 20, the two instruments 121R and 121L, which are surgical instruments, have a configuration that can be directly operated by the surgeon.

[0141] Furthermore, the instrument 121R is provided with at least a shaft portion 22R, an end effector portion 23R, a shaft sensor 61R, and an end effector sensor 62R. Similarly, the instrument 121L is provided with at least a shaft portion 22L, an end effector portion 23L, a shaft sensor 61L, and an end effector sensor 62L.

[0142] (Modifications of the first embodiment)

[0143] Next, refer to Figure 11 The force display device 50A, a variation of the first embodiment of the present disclosure, will be described.

[0144] The processing of the surgical instrument detection area in the basic configuration of the force display device 50A in this modified example differs from that in the first embodiment. Therefore, in this embodiment, refer to... Figure 11 The treatment of the area for detecting surgical instruments is explained, and other explanations are omitted.

[0145] like Figure 11 As shown, the force display device 50A of this modified example is an information processing device such as a processor, microcomputer, personal computer, and server, which has a CPU (central processing unit), ROM, RAM, input / output interface, etc.

[0146] The program stored in the aforementioned storage device such as ROM enables the CPU, ROM, RAM, and input / output interface to work together, and functions as at least the image acquisition unit 51, the force acquisition unit 52, the generation unit 53, the detection unit 54A, the overlay unit 55, the output unit 56, and the storage unit 57A.

[0147] The storage unit 57A stores at least one of the colors and patterns of the devices 121R and 121L. In this embodiment, the storage unit 57A is described as storing the colors assigned to the shaft portion 22R, the end effector portion 23R, the shaft portion 22L, and the end effector portion 23L as an example. Furthermore, the storage unit 57A can also store combinations of colors and patterns of the devices 121R and 121L, and can also store patterns.

[0148] The colors of instruments 121R and 121L are preferably complementary colors to the color of the affected area. For example, green, the complementary color to the color of blood, is preferred. Blood color includes arterial blood color and venous blood color.

[0149] The design of instruments 121R and 121L is preferably a geometric design that is easy to distinguish from the affected area. For example, a linear geometric design or a design in which the same shape appears repeatedly is preferred.

[0150] The detection unit 54A is a computation unit that detects the surgical instrument area based on the acquired image information and the stored color information of the instruments 121R and 121L. As for the image analysis processing for detecting the surgical instrument area based on color information, the processing method commonly used when detecting a specified area based on color information can be employed.

[0151] If the colors of instrument 121R and instrument 121L are different, it is easy to detect the surgical instrument area of ​​instrument 121R and the surgical instrument area of ​​instrument 121L as different areas.

[0152] Furthermore, if the color of the shaft portion 22R and the color of the end effector portion 23R are different, it is easy to detect the first surgical instrument area 27R and the second surgical instrument area 28R as different areas. Similarly, if the color of the shaft portion 22L and the color of the end effector portion 23L are different, it is easy to detect the first surgical instrument area 27L and the second surgical instrument area 28L as different areas.

[0153] Next, regarding the method for displaying force in the force display device 50A having the above configuration, except for the difference in image analysis and processing of the area for detecting surgical instruments, it is the same as the method for displaying force in the first embodiment, so this description is omitted.

[0154] According to the force-sensing display device 50A described above, the first surgical instrument area 27R, the second surgical instrument area 28R, the first surgical instrument area 27L, and the second surgical instrument area 28L are detected based on the colors of the instruments 121R and 121L. This facilitates the detection of the first surgical instrument area 27R, the second surgical instrument area 28R, the first surgical instrument area 27L, and the second surgical instrument area 28L. Furthermore, it easily improves the detection accuracy of the first surgical instrument area 27R, the second surgical instrument area 28R, the first surgical instrument area 27L, and the second surgical instrument area 28L.

[0155] Furthermore, the technical scope of this invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of this invention.

Claims

1. A force-sensing display device, characterized in that, The settings are as follows: An image acquisition unit acquires imaging information output from an imaging unit that captures images of the affected area; The force sensing acquisition unit acquires force sensing information, which includes information on the magnitude and direction of the force exerted on at least one surgical instrument treating the affected area. The generation unit generates a force display based on the force information. The force display is a display form displayed on the display unit and includes a force display indicating the magnitude of the force and a direction display indicating the direction of the force. The detection unit detects a surgical instrument area based on the captured information; the surgical instrument area is the area in the image displayed on the display unit where the surgical instrument is displayed. as well as The overlay section generates display information, which is information that causes the force sensor display to be overlaid onto the surgical instrument area and displayed on the display section. The force information includes information on the magnitude and direction of a first force acting on the rod-shaped shaft portion of the surgical instrument. The generating unit generates a first force display, which includes a first force display indicating the magnitude of the first force and a first direction display indicating the direction of the first force. The detection unit will detect the area in the image displayed on the display unit that contains the shaft portion as the first surgical instrument area. The overlay unit generates the display information, which is information that causes the first force sensor display to be overlaid onto the first surgical instrument area and displayed on the display unit. Furthermore, the force information includes information on the magnitude and direction of a second force applied to the end effector, which is located at the end of the shaft and used to treat the affected area. The generating unit generates a second force display, which includes a second force display indicating the magnitude of the second force and a second direction display indicating the direction of the second force. The detection unit will detect the area in the image displayed on the display unit where the end effector unit is shown as the second surgical instrument area. The overlay section generates the display information, which is information that causes the second force sensor display to be overlaid on the second surgical instrument area and displayed on the display section.

2. The force display device according to claim 1, characterized in that, The generating unit generates the force display, which includes a force display that uses at least one of color type, color brightness, and translucency to represent the magnitude of the force.

3. The force display device according to claim 1, characterized in that, The generating unit generates the force display, which includes a force display that uses numerical values ​​to represent the magnitude of the force.

4. The force display device according to claim 3, characterized in that, The generating unit generates the force display, which includes a force display that uses the size of the displayed numerical value to represent the magnitude of the force.

5. The force-sensing display device according to any one of claims 1 to 4, characterized in that, It also includes a storage unit that stores at least one of the colors and styles of the surgical instruments. The detection unit detects the surgical instrument area in the image displayed on the display unit based on at least one of the colors and patterns of the surgical instruments stored in the storage unit.

6. A force-sensing display method, comprising a force-sensing display device having an image acquisition unit, a force-sensing acquisition unit, a generation unit, a detection unit, and an overlay unit, characterized in that it includes: The image acquisition step involves the image acquisition unit acquiring the imaging information output from the imaging unit that captures the affected area; The force sensing acquisition step involves the force sensing acquisition unit acquiring force sensing information, which includes information on the magnitude and direction of the force exerted on at least one surgical instrument used to treat the affected area. In the generation step, the generation unit generates a force display based on the force information. The force display is a display form displayed on the display unit and includes a force display indicating the magnitude of the force and a direction display indicating the direction of the force. In the detection step, the detection unit detects the surgical instrument area based on the captured information. The surgical instrument area is the area in the image displayed on the display unit where the surgical instrument is displayed. as well as In the overlay step, the overlay unit generates display information, which is information that causes the force sensor display to be overlaid onto the surgical instrument area and displayed on the display unit. The force information includes information on the magnitude and direction of a first force acting on the rod-shaped shaft portion of the surgical instrument. In the generation step, a first force display is generated, which includes a first force display indicating the magnitude of the first force and a first direction display indicating the direction of the first force. In the detection step, the area in the image displayed on the display unit that shows the shaft portion is detected as the first surgical instrument area. The display information is generated in the overlay step. This display information is information that causes the first force sensor display to be overlaid onto the first surgical instrument area and displayed on the display unit. Furthermore, the force information includes information on the magnitude and direction of a second force applied to the end effector, which is located at the end of the shaft and used to treat the affected area. In the generation step, a second force display is generated, which includes a second force display indicating the magnitude of the second force and a second direction display indicating the direction of the second force. In the detection step, the area in the image displayed on the display unit that shows the end effector is detected as the second surgical instrument area. The display information is generated in the overlay step. This display information is information that causes the second force sensory display to be overlaid onto the second surgical instrument area and displayed on the display unit.

7. A computer program product, characterized in that, The computer stores programs for enabling the computer to perform the following functions: Image acquisition function: acquires the image information output from the imaging unit that captures the image of the affected area; Force sensing acquisition function, which acquires force sensing information, including information on the magnitude and direction of the force exerted on at least one surgical instrument treating the affected area; The generation function generates a force display based on the force information. The force display is a display form shown on the display unit and includes a force display indicating the magnitude of the force and a direction display indicating the direction of the force. The detection function detects a surgical instrument area based on the captured information; the surgical instrument area is the area in the image displayed on the display unit where the surgical instruments are displayed; and The overlay function generates display information, which is information that overlays the force sensor display onto the surgical instrument area and displays it on the display unit. The force information includes information on the magnitude and direction of a first force acting on the rod-shaped shaft portion of the surgical instrument. The generation function generates a first force display, which includes a first force display indicating the magnitude of the first force and a first direction display indicating the direction of the first force. The detection function identifies the area in the image displayed on the display unit that contains the shaft portion as the first surgical instrument area. The overlay function generates the display information, which is information that causes the first force sensor display to be overlaid onto the first surgical instrument area and displayed on the display unit. Furthermore, the force information includes information on the magnitude and direction of a second force applied to the end effector, which is located at the end of the shaft and used to treat the affected area. The generation function generates a second force display, which includes a second force display indicating the magnitude of the second force and a second direction display indicating the direction of the second force. The detection function identifies the area in the image displayed on the display unit where the end effector is located as the second surgical instrument area. The overlay function generates the display information, which is information that overlays the second force sensor display onto the second surgical instrument area and displays it on the display unit.

Images