Medical arm control system, medical arm control method, and program

Abstract

To provide a medical arm control system capable of generating in real time autonomous action control information for a robot arm device, the autonomous action control information being suitable for various body's internal environments.SOLUTION: A medical art control system includes: a control information generation unit generating autonomous action control information for operating a medical arm autonomously on the basis of external input information; a simulation unit performing an action simulation of the medical arm; and a correction unit correcting the autonomous action control information in real time on the basis of a result of the action simulation of the medical arm.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a medical arm control system, a medical arm control method, and a program. [Background technology]

[0002] In recent years, in endoscopic surgery, an endoscope is used to capture images of the patient's abdominal cavity, and the surgery is performed while the captured images are displayed on a display. For example, Patent Document 1 listed below discloses a technology for linking control of an arm that supports an endoscope with control of the electronic zoom of the endoscope. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 145475 Summary of the Invention [Problem to be solved by the invention]

[0004] Recently, in the field of medical observation systems, development has been progressing to autonomously operate the robot arm device that supports the endoscope. For example, there are attempts to autonomously control the movement of the endoscope (scope work) during surgery using a robot arm device, allowing the surgeon to perform procedures using surgical tools in that environment.

[0005] However, because the internal environment of each patient is different, even if multiple cases are studied in advance to enable autonomous operation, unknown environments will inevitably exist during actual surgery.

[0006] Therefore, this disclosure proposes a medical arm control system, a medical arm control method, and a program that are capable of generating autonomous operation control information for a robot arm device in real time, suitable for various internal body environments. [Means for solving the problem]

[0007] According to the present disclosure, there is provided a medical arm control system comprising: a control information generation unit that generates autonomous operation control information for autonomously operating a medical arm based on external input information; a simulation unit that performs an operation simulation of the medical arm; and a correction unit that corrects the autonomous operation control information in real time based on the results of the operation simulation of the medical arm.

[0008] Furthermore, according to the present disclosure, there is provided a medical arm control method, which includes generating autonomous operation control information for autonomously operating a medical arm based on external input information by a medical arm control device, simulating the operation of the medical arm, and correcting the autonomous operation control information in real time based on the results of the operation simulation of the medical arm.

[0009] Furthermore, according to the present disclosure, there is provided a program that causes a computer to function as a control information generation unit that generates autonomous movement control information for autonomously operating a medical arm based on external input information, a simulation unit that performs a movement simulation of the medical arm, and a correction unit that corrects the autonomous movement control information in real time based on the results of the movement simulation of the medical arm. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology disclosed herein can be applied. [Figure 2] 2 is a block diagram showing an example of the functional configuration of a camera head and a CCU (Camera Control Unit) shown in FIG. 1. FIG. [Figure 3] 1 is a schematic diagram illustrating a configuration of an oblique-viewing mirror according to an embodiment of the present disclosure. [Figure 4] 1 is a block diagram showing an example of the configuration of a medical observation system according to an embodiment of the present disclosure. [Figure 5]FIG. 2 is a block diagram illustrating an example of the configuration of a learning model generation unit according to an embodiment of the present disclosure. [Figure 6] 1 is a flowchart of a control method at the learning model generation stage according to an embodiment of the present disclosure. [Figure 7] FIG. 2 is a block diagram illustrating an example of a configuration of an autonomous operation execution unit according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a block diagram illustrating an example of a configuration of a presentation device according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart (part 1) of a control method at an autonomous operation execution stage according to an embodiment of the present disclosure. [Figure 10] 10 is a sub-flowchart of step S200 shown in FIG. 9. [Figure 11] FIG. 1 is an explanatory diagram (part 1) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 12] FIG. 10 is an explanatory diagram (part 2) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 13] FIG. 10 is an explanatory diagram (part 3) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 14] FIG. 10 is an explanatory diagram (part 4) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 15] FIG. 5 is an explanatory diagram (part 5) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 16] FIG. 6 is an explanatory diagram (part 6) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 17] FIG. 7 is an explanatory diagram (part 7) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 18] FIG. 8 is an explanatory diagram (part 8) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 19] FIG. 9 is an explanatory diagram (part 9) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 20] FIG. 10 is a tenth explanatory diagram for explaining details of a control method according to an embodiment of the present disclosure. [Figure 21] 10 is a sub-flowchart of step S300 shown in FIG. 9. [Figure 22] FIG. 11 is an explanatory diagram (part 11) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 23] FIG. 12 is a twelfth explanatory diagram for explaining details of a control method according to an embodiment of the present disclosure. [Figure 24] FIG. 13 is an explanatory diagram (part 13) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 25] FIG. 14 is an explanatory diagram (part 14) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 26] 10 is a flowchart (part 2) of a control method in an autonomous operation execution stage according to an embodiment of the present disclosure. [Figure 27] FIG. 15 is an explanatory diagram (part 15) for explaining details of a control method according to an embodiment of the present disclosure. [Figure 28] FIG. 1 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an autonomous operation execution unit according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. Furthermore, in this specification and the drawings, multiple components having substantially the same or similar functional configurations may be distinguished by adding different letters after the same reference numeral. However, when there is no particular need to distinguish between multiple components having substantially the same or similar functional configurations, only the same reference numerals will be used.

[0012] The explanation will be given in the following order. 1. Example of the configuration of the endoscopic surgery system 5000 1.1 Schematic configuration of the endoscopic surgery system 5000 1.2 Detailed configuration example of support arm device 5027 1.3 Detailed configuration example of light source device 5043 1.4 Detailed configuration example of camera head 5005 and CCU 5039 1.5 Example of Endoscope 5001 Configuration 2. Medical Observation System 3. Background leading to the creation of the embodiments of the present disclosure 4. Implementation form 4.1 Detailed configuration example of the learning model generation unit 100 4.2 Control method in the learning model generation stage 4.3 Detailed configuration example of the autonomous operation execution unit 200 4.4 Control method during autonomous operation execution 4.5 Variations 5. Hardware Configuration 6. Supplementary Information

[0013] <<1. Configuration Example of Endoscopic Surgery System 5000>> 1.1 Schematic configuration of the endoscopic surgery system 5000 Before describing the details of the embodiments of the present disclosure, a schematic configuration of an endoscopic surgery system 5000 to which the technology of the present disclosure can be applied will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the schematic configuration of an endoscopic surgery system 5000 to which the technology of the present disclosure can be applied. FIG. 1 illustrates a state in which an operator (doctor) 5067 is performing surgery on a patient 5071 on a patient bed 5069 using the endoscopic surgery system 5000. As shown in FIG. 1, the endoscopic surgery system 5000 includes an endoscope 5001, other surgical instruments 5017, a support arm device 5027 that supports the endoscope 5001, and a cart 5037 on which various devices for endoscopic surgery are mounted. The details of the endoscopic surgery system 5000 will be described below.

[0014] (Surgical Tool 5017) In endoscopic surgery, instead of cutting the abdominal wall and opening the abdomen, for example, a plurality of cylindrical drilling instruments called trocars 5025a to 5025d are punctured into the abdominal wall. Then, a lens barrel 5003 of an endoscope 5001 and other surgical instruments 5017 are inserted into the body cavity of a patient 5071 through the trocars 5025a to 5025d. In the example shown in FIG. 1 , as the other surgical instruments 5017, an insufflation tube 5019, an energy treatment instrument 5021, and forceps 5023 are inserted into the body cavity of the patient 5071. The energy treatment instrument 5021 is a treatment instrument that performs incision and dissection of tissue, sealing of blood vessels, etc., using high-frequency current or ultrasonic vibration. However, the surgical instrument 5017 shown in FIG. 1 is merely an example, and various surgical instruments generally used in endoscopic surgery, such as a suction cup or a retractor, can be used as the surgical instrument 5017.

[0015] (Support arm device 5027) The support arm device 5027 has an arm portion 5031 extending from a base portion 5029. In the example shown in Fig. 1, the arm portion 5031 is composed of joints 5033a, 5033b, and 5033c and links 5035a and 5035b, and is driven under control of an arm control device 5045. The arm portion 5031 supports the endoscope 5001, and controls the position and posture of the endoscope 5001. This makes it possible to stably fix the position of the endoscope 5001.

[0016] (Endoscope 5001) The endoscope 5001 is composed of a lens barrel 5003, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 5071, and a camera head 5005 connected to the base end of the lens barrel 5003. In the example shown in Fig. 1, the endoscope 5001 is configured as a so-called rigid lens barrel having a rigid lens barrel 5003, but the endoscope 5001 may also be configured as a so-called flexible lens barrel having a flexible lens barrel 5003, and is not particularly limited in the embodiments of the present disclosure.

[0017] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 5003. A light source device 5043 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 5003, and is irradiated via the objective lens toward an observation target inside the body cavity of the patient 5071. Note that in the embodiment of the present disclosure, the endoscope 5001 may be a forward-viewing endoscope or an oblique-viewing endoscope, and is not particularly limited.

[0018] An optical system and a light receiving element are provided inside the camera head 5005, and reflected light (observation light) from the observation object is collected on the light receiving element by the optical system. The observation light is photoelectrically converted by the light receiving element to generate an electrical signal corresponding to the observation light, i.e., a pixel signal corresponding to the observation image. The pixel signal is transmitted as RAW data to a camera control unit (CCU) 5039. The camera head 5005 is equipped with a function for adjusting the magnification and focal length by appropriately driving the optical system.

[0019] Note that, for example, to support stereoscopic vision (3D display), a plurality of light receiving elements may be provided in the camera head 5005. In this case, a plurality of relay optical systems are provided inside the lens barrel 5003 to guide observation light to each of the plurality of light receiving elements.

[0020] (Various devices mounted on the cart) First, under the control of the CCU 5039, the display device 5041 displays an image based on an image signal generated by image processing performed on a pixel signal by the CCU 5039. If the endoscope 5001 is compatible with high-resolution imaging such as 4K (3840 horizontal pixels × 2160 vertical pixels) or 8K (7680 horizontal pixels × 4320 vertical pixels) and / or is compatible with 3D display, a display device capable of displaying the corresponding high resolution and / or 3D display is used as the display device 5041. Furthermore, multiple display devices 5041 with different resolutions and sizes may be provided depending on the application.

[0021] Furthermore, an image of the area to be operated on inside the body cavity of the patient 5071 photographed by the endoscope 5001 is displayed on the display device 5041. The surgeon 5067 can perform treatment such as excising the affected area using the energy treatment tool 5021 and forceps 5023 while viewing the image of the area to be operated on displayed on the display device 5041 in real time. Although not shown in the drawings, the insufflation tube 5019, the energy treatment tool 5021, and the forceps 5023 may be supported by the surgeon 5067 or an assistant during surgery.

[0022] The CCU 5039 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and can comprehensively control the operations of the endoscope 5001 and the display device 5041. Specifically, the CCU 5039 performs various image processing, such as development processing (demosaic processing), on pixel signals received from the camera head 5005 in order to display an image based on the pixel signals. The CCU 5039 then provides the image signals generated by the image processing to the display device 5041. The CCU 5039 also transmits control signals to the camera head 5005 to control its driving. The control signals can include information regarding imaging conditions, such as magnification and focal length.

[0023] The light source device 5043 is configured from a light source such as an LED (Light Emitting Diode), and supplies the endoscope 5001 with irradiation light when photographing the operation site.

[0024] The arm control device 5045 is configured by a processor such as a CPU, and operates according to a predetermined program to control the driving of the arm portion 5031 of the support arm device 5027 according to a predetermined control method.

[0025] The input device 5047 is an input interface for the endoscopic surgery system 5000. The surgeon 5067 can input various pieces of information and instructions to the endoscopic surgery system 5000 via the input device 5047. For example, the surgeon 5067 can input various pieces of information related to the surgery, such as physical information about the patient and information about the surgical procedure, via the input device 5047. Furthermore, for example, the surgeon 5067 can input, via the input device 5047, an instruction to drive the arm unit 5031, an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 5001, an instruction to drive the energy treatment tool 5021, etc. Note that the type of the input device 5047 is not limited, and the input device 5047 may be any known input device. For example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057, and / or a lever may be used as the input device 5047. For example, when a touch panel is used as the input device 5047, the touch panel may be provided on the display surface of the display device 5041.

[0026] Alternatively, the input device 5047 may be a device worn by the surgeon 5067, such as a glasses-type wearable device or an HMD (Head Mounted Display). In this case, various inputs are made according to the gestures and line of sight of the surgeon 5067 detected by these devices. The input device 5047 may also include a camera capable of detecting the movements of the surgeon 5067, and various inputs may be made according to the gestures and line of sight of the surgeon 5067 detected from images captured by the camera. The input device 5047 may also include a microphone capable of collecting the voice of the surgeon 5067, and various inputs may be made by voice via the microphone. In this way, the input device 5047 is configured to be able to input various information contactlessly, thereby enabling a user (e.g., the surgeon 5067) in a clean area to operate equipment in an unclean area in a contactless manner. Furthermore, the surgeon 5067 can operate equipment without removing his or her hands from the surgical tools he or she is holding, thereby improving convenience for the surgeon 5067.

[0027] The treatment tool control device 5049 controls the driving of an energy treatment tool 5021 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 5051 sends gas into the body cavity of the patient 5071 via an insufflation tube 5019 to ensure a clear field of view for the endoscope 5001 and to ensure a working space for the surgeon. The recorder 5053 is a device capable of recording various types of information related to the surgery. The printer 5055 is a device capable of printing various types of information related to the surgery in various formats such as text, images, or graphs.

[0028] <1.2 Detailed configuration example of the support arm device 5027> Next, an example of the detailed configuration of the support arm device 5027 will be described. The support arm device 5027 has a base 5029 serving as a base and an arm 5031 extending from the base 5029. In the example shown in FIG. 1, the arm 5031 is composed of multiple joints 5033a, 5033b, and 5033c and multiple links 5035a and 5035b connected by the joint 5033b. However, for simplicity, FIG. 1 illustrates a simplified configuration of the arm 5031. Specifically, the shapes, number, and arrangement of the joints 5033a to 5033c and the links 5035a and 5035b, as well as the directions of the rotation axes of the joints 5033a to 5033c, can be set appropriately so that the arm 5031 has the desired degrees of freedom. For example, the arm 5031 can be preferably configured to have six or more degrees of freedom. This allows the endoscope 5001 to be moved freely within the movable range of the arm portion 5031, making it possible to insert the lens barrel 5003 of the endoscope 5001 into the body cavity of the patient 5071 from the desired direction.

[0029] The joints 5033a to 5033c are provided with actuators, and the joints 5033a to 5033c are configured to be rotatable around predetermined rotation axes by driving the actuators. The driving of the actuators is controlled by an arm control device 5045, thereby controlling the rotation angles of the joints 5033a to 5033c and controlling the driving of the arm 5031. This makes it possible to control the position and attitude of the endoscope 5001. In this case, the arm control device 5045 can control the driving of the arm 5031 by various known control methods, such as force control or position control.

[0030] For example, the surgeon 5067 may appropriately input an operation via the input device 5047 (including the foot switch 5057), and the arm control device 5045 may appropriately control the drive of the arm unit 5031 in accordance with the operation input, thereby controlling the position and posture of the endoscope 5001. The arm unit 5031 may be operated in a so-called master-slave manner. In this case, the arm unit 5031 (slave) may be remotely controlled by the surgeon 5067 via the input device 5047 (master console) installed in a location away from the operating room or in the operating room.

[0031] Generally, in endoscopic surgery, the endoscope 5001 is supported by a doctor called a scopist. In contrast, in the embodiment of the present disclosure, by using the support arm device 5027, the position of the endoscope 5001 can be more reliably fixed without manual intervention, making it possible to obtain stable images of the surgical site and perform the surgery smoothly.

[0032] It should be noted that the arm control device 5045 does not necessarily have to be provided on the cart 5037. Furthermore, the arm control device 5045 does not necessarily have to be one device. For example, an arm control device 5045 may be provided on each of the joints 5033a to 5033c of the arm section 5031 of the support arm device 5027, and the drive control of the arm section 5031 may be realized by a plurality of arm control devices 5045 working together.

[0033] <1.3 Detailed configuration example of light source device 5043> Next, an example of the detailed configuration of the light source device 5043 will be described. The light source device 5043 supplies illumination light to the endoscope 5001 when photographing the surgical site. The light source device 5043 is configured from a white light source formed, for example, by an LED, a laser light source, or a combination thereof. In this case, if the white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, allowing the light source device 5043 to adjust the white balance of the captured image. In this case, it is also possible to irradiate the object of observation with laser light from each of the RGB laser light sources in a time-division manner and control the drive of the light receiving elements of the camera head 5005 in synchronization with the irradiation timing, thereby capturing images corresponding to each of the RGB colors in a time-division manner. According to this method, a color image can be obtained without providing a color filter to the light receiving elements.

[0034] Furthermore, the light source device 5043 may be controlled to change the intensity of light it outputs at predetermined time intervals. By controlling the driving of the light receiving elements of the camera head 5005 in synchronization with the timing of the change in the light intensity to acquire images in a time-division manner and combining the images, it is possible to generate an image with a high dynamic range that is free from so-called blocked-up shadows and blown-out highlights.

[0035] The light source device 5043 may also be configured to supply light in a predetermined wavelength band corresponding to special light observation. Special light observation, for example, utilizes the wavelength dependency of light absorption in body tissues to irradiate light with a narrower band than the light irradiated during normal observation (i.e., white light), thereby capturing high-contrast images of specific tissues, such as blood vessels on the surface of mucous membranes, known as narrow-band imaging. Alternatively, special light observation may involve fluorescence observation, in which images are obtained using fluorescence generated by irradiating excitation light. Fluorescence observation may involve irradiating excitation light onto body tissues and observing the fluorescence from the tissue (autofluorescence observation), or irradiating the tissues with excitation light corresponding to the fluorescent wavelength of a reagent such as indocyanine green (ICG) to obtain a fluorescent image. The light source device 5043 may be configured to supply narrow-band light and / or excitation light corresponding to such special light observation.

[0036] <1.4 Detailed configuration example of camera head 5005 and CCU 5039> Next, an example of the detailed configuration of the camera head 5005 and the CCU 5039 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the camera head 5005 and the CCU 5039 shown in Fig. 1.

[0037] 2, the camera head 5005 has, as its functions, a lens unit 5007, an imaging unit 5009, a drive unit 5011, a communication unit 5013, and a camera head control unit 5015. The CCU 5039 has, as its functions, a communication unit 5059, an image processing unit 5061, and a control unit 5063. The camera head 5005 and the CCU 5039 are connected via a transmission cable 5065 to enable bidirectional communication.

[0038] First, the functional configuration of the camera head 5005 will be described. The lens unit 5007 is an optical system provided at the connection portion with the lens barrel 5003. Observation light taken in from the tip of the lens barrel 5003 is guided to the camera head 5005 and enters the lens unit 5007. The lens unit 5007 is configured by combining multiple lenses including a zoom lens and a focus lens. The optical characteristics of the lens unit 5007 are adjusted so that the observation light is focused on the light receiving surface of the light receiving element of the imaging section 5009. In addition, the zoom lens and the focus lens are configured so that their positions on the optical axis can be moved to adjust the magnification and focus of the captured image.

[0039] The imaging unit 5009 is composed of a light-receiving element and is arranged after the lens unit 5007. Observation light that has passed through the lens unit 5007 is collected on the light-receiving surface of the light-receiving element, and pixel signals corresponding to the observation image are generated by photoelectric conversion. The pixel signals generated by the imaging unit 5009 are provided to the communication unit 5013.

[0040] The light receiving element constituting the imaging unit 5009 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor that has a Bayer array and is capable of color imaging. The light receiving element may be capable of capturing high-resolution images of, for example, 4K or higher. Obtaining high-resolution images of the surgical site allows the surgeon 5067 to grasp the state of the surgical site in more detail, enabling the surgery to proceed more smoothly.

[0041] Furthermore, the light receiving elements constituting the imaging unit 5009 may be configured to have a pair of light receiving elements for respectively acquiring pixel signals for the right eye and the left eye corresponding to 3D display (stereo system). 3D display enables the surgeon 5067 to more accurately grasp the depth of the biological tissue at the surgical site and grasp the distance to the biological tissue. Note that when the imaging unit 5009 is configured as a multi-plate type, multiple lens units 5007 may be provided corresponding to the respective light receiving elements.

[0042] Furthermore, the imaging unit 5009 does not necessarily have to be provided in the camera head 5005. For example, the imaging unit 5009 may be provided inside the lens barrel 5003, immediately after the objective lens.

[0043] The driving section 5011 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 5007 by a predetermined distance along the optical axis under the control of the camera head control section 5015. This allows the magnification and focus of the image captured by the imaging section 5009 to be adjusted appropriately.

[0044] The communication unit 5013 is configured by a communication device for transmitting and receiving various information to and from the CCU 5039. The communication unit 5013 transmits pixel signals obtained from the imaging unit 5009 as RAW data to the CCU 5039 via the transmission cable 5065. At this time, in order to display the captured image of the surgical site with low latency, it is preferable that the pixel signals be transmitted by optical communication. This is because, during surgery, the surgeon 5067 performs surgery while observing the condition of the affected area using the captured image, and for a safer and more reliable surgery, it is necessary that moving images of the surgical site be displayed as real-time as possible. When optical communication is performed, the communication unit 5013 is provided with a photoelectric conversion module that converts electrical signals into optical signals. The pixel signals are converted into optical signals by the photoelectric conversion module and then transmitted to the CCU 5039 via the transmission cable 5065.

[0045] The communication unit 5013 also receives control signals from the CCU 5039 for controlling the operation of the camera head 5005. The control signals include information related to imaging conditions, such as information specifying the frame rate of an image to be captured, information specifying an exposure value during imaging, and / or information specifying the magnification and focus of an image to be captured. The communication unit 5013 provides the received control signals to the camera head control unit 5015. The control signals from the CCU 5039 may also be transmitted by optical communication. In this case, the communication unit 5013 is provided with a photoelectric conversion module that converts optical signals into electrical signals, and the control signals are converted into electrical signals by the photoelectric conversion module and then provided to the camera head control unit 5015.

[0046] The image capturing conditions such as the frame rate, exposure value, magnification, and focus are automatically set by the control unit 5063 of the CCU 5039 based on the acquired pixel signals. That is, the endoscope 5001 is equipped with so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0047] The camera head control unit 5015 controls the driving of the camera head 5005 based on a control signal received from the CCU 5039 via the communication unit 5013. For example, the camera head control unit 5015 controls the driving of the light receiving element of the imaging unit 5009 based on information specifying the frame rate of the captured image and / or information specifying the exposure during imaging. Also, for example, the camera head control unit 5015 appropriately moves the zoom lens and focus lens of the lens unit 5007 via the drive unit 5011 based on information specifying the magnification and focus of the captured image. The camera head control unit 5015 may further have a function of storing information for identifying the lens barrel 5003 and the camera head 5005.

[0048] Incidentally, by arranging the components such as the lens unit 5007 and the imaging unit 5009 in a sealed structure that is highly airtight and waterproof, the camera head 5005 can be made resistant to autoclave sterilization.

[0049] Next, the functional configuration of the CCU 5039 will be described. The communication unit 5059 is configured by a communication device for transmitting and receiving various information to and from the camera head 5005. The communication unit 5059 receives pixel signals transmitted from the camera head 5005 via the transmission cable 5065. At this time, as described above, the pixel signals may be preferably transmitted by optical communication. In this case, in order to support optical communication, the communication unit 5059 is provided with an optoelectric conversion module that converts optical signals into electrical signals. The communication unit 5059 provides the pixel signals converted into electrical signals to the image processing unit 5061.

[0050] Furthermore, the communication unit 5059 transmits to the camera head 5005 a control signal for controlling the driving of the camera head 5005. This control signal may also be transmitted by optical communication.

[0051] The image processing unit 5061 performs various types of image processing on pixel signals, which are RAW data transmitted from the camera head 5005. The image processing includes various known signal processing such as development processing, high-quality image processing (band enhancement processing, super-resolution processing, NR (Noise Reduction) processing, and / or image stabilization processing), and / or enlargement processing (electronic zoom processing), etc. The image processing unit 5061 also performs detection processing on pixel signals to perform AE, AF, and AWB.

[0052] The image processing unit 5061 is configured with a processor such as a CPU or GPU, and the processor operates in accordance with a predetermined program to perform the image processing and detection processing described above. Note that if the image processing unit 5061 is configured with multiple GPUs, the image processing unit 5061 divides information related to pixel signals appropriately and performs image processing in parallel using these multiple GPUs.

[0053] The control unit 5063 performs various controls related to the imaging of the surgical site by the endoscope 5001 and the display of the captured image. For example, the control unit 5063 generates a control signal for controlling the driving of the camera head 5005. At this time, if the imaging conditions have been input by the surgeon 5067, the control unit 5063 generates the control signal based on the input by the surgeon 5067. Alternatively, if the endoscope 5001 is equipped with an AE function, an AF function, and an AWB function, the control unit 5063 appropriately calculates the optimal exposure value, focal length, and white balance according to the result of detection processing by the image processing unit 5061, and generates the control signal.

[0054] The control unit 5063 also displays an image of the surgical site on the display device 5041 based on the image signal generated by the image processing unit 5061 after image processing. At this time, the control unit 5063 recognizes various objects in the surgical site image using various image recognition technologies. For example, the control unit 5063 can recognize surgical tools such as forceps, specific biological parts, bleeding, mist generated when using the energy treatment tool 5021, and the like, by detecting the shape and color of the edges of objects included in the surgical site image. When displaying the image of the surgical site on the display device 5041, the control unit 5063 uses the recognition results to superimpose various surgical support information on the image of the surgical site. The superimposed surgical support information and its presentation to the surgeon 5067 enable the surgery to proceed more safely and reliably.

[0055] The transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electric signal cable for communication of electric signals, an optical fiber for optical communication, or a composite cable of these.

[0056] In the illustrated example, communication is performed by wire using the transmission cable 5065, but communication between the camera head 5005 and the CCU 5039 may be performed wirelessly. When communication between them is performed wirelessly, there is no need to lay the transmission cable 5065 in the operating room, which can eliminate the situation where the transmission cable 5065 interferes with the movement of medical staff in the operating room.

[0057] <1.5 Configuration example of endoscope 5001> Next, a basic configuration of an oblique mirror will be described as an example of an endoscope 5001 with reference to Fig. 3. Fig. 3 is a schematic diagram showing the configuration of an oblique mirror 4100 according to an embodiment of the present disclosure.

[0058] 3, the oblique mirror 4100 is attached to the tip of the camera head 4200. The oblique mirror 4100 corresponds to the lens barrel 5003 described in FIGS. 1 and 2, and the camera head 4200 corresponds to the camera head 5005 described in FIGS. 1 and 2. The oblique mirror 4100 and the camera head 4200 are rotatable independently of each other. An actuator is provided between the oblique mirror 4100 and the camera head 4200, similar to each of the joints 5033a, 5033b, and 5033c, and the oblique mirror 4100 rotates relative to the camera head 4200 by driving the actuator.

[0059] The oblique scope 4100 is supported by a support arm device 5027. The support arm device 5027 has the function of holding the oblique scope 4100 on behalf of the scopist and moving the oblique scope 4100 by operation of the surgeon or assistant so that a desired area can be observed.

[0060] It should be noted that in the embodiment of the present disclosure, the endoscope 5001 is not limited to the oblique-viewing scope 4100. For example, the endoscope 5001 may be a forward-viewing scope (not shown) that captures the area in front of the tip of the endoscope, and may further have a function of extracting an image from a wide-angle image captured by the endoscope (wide-angle / extraction function). Furthermore, for example, the endoscope 5001 may be an endoscope (not shown) with a tip bending function that allows the tip of the endoscope to be freely bent in accordance with the operation of the surgeon 5067, thereby changing the field of view. Furthermore, for example, the endoscope 5001 may be an endoscope (not shown) with a multi-directional simultaneous imaging function that incorporates multiple camera units with different fields of view at the tip of the endoscope, allowing different images to be obtained by each camera.

[0061] The above describes an example of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied. Note that although the endoscopic surgery system 5000 has been described as an example here, systems to which the technology according to the present disclosure can be applied are not limited to this example. For example, the technology according to the present disclosure may be applied to a microsurgical system.

[0062] <<2. Medical Observation System>> Furthermore, the configuration of a medical observation system 1 according to an embodiment of the present disclosure, which can be combined with the above-described endoscopic surgery system 5000, will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of the medical observation system 1 according to an embodiment of the present disclosure. As shown in Fig. 4, the medical observation system 1 mainly includes a robot arm device 10, an imaging unit 12, a light source unit 13, a control unit 20, a presentation device 40, and a storage unit 60. Each functional unit included in the medical observation system 1 will be described below.

[0063] First, before explaining the details of the configuration of the medical observation system 1, we will explain an overview of the processing of the medical observation system 1. In the medical observation system 1, first, an image of the inside of the abdominal cavity of the patient is captured to recognize the environment inside the abdominal cavity, and the robot arm device 10 can be driven based on the recognition result of the environment inside the abdominal cavity.

[0064] (Robot arm device 10) The robot arm device 10 has an arm section 11 (multi-joint arm) that is a multi-link structure composed of multiple joints and multiple links, and controls the position and posture of a tip unit provided at the tip of the arm section by driving the arm section within a movable range. The robot arm device 10 corresponds to the support arm device 5027 shown in FIG.

[0065] The robot arm device 10 may have, for example, a CCU 5039 shown in FIG. 2, an electronic cut-out control unit (not shown) that cuts out a predetermined area from an image of the object to be photographed received from the CCU 5039 and outputs the area to a GUI generation unit (described later), an attitude control unit (not shown) that controls the position and attitude of the arm unit 11, and a GUI generation unit (not shown) that generates image data by performing various processes on the image cut out from the electronic cut-out control unit.

[0066] In the robot arm device 10 according to the embodiment of the present disclosure, the electronic degree of freedom for changing the line of sight by cropping a captured image (wide-angle / cropping function) and the degree of freedom provided by the actuator of the arm unit 11 are all treated as degrees of freedom of the robot. This makes it possible to realize motion control that links the electronic degree of freedom for changing the line of sight with the degree of freedom of the joint provided by the actuator.

[0067] Specifically, the arm unit 11 is a multi-link structure composed of a plurality of joints and a plurality of links, and its drive is controlled by the arm control unit 23, which will be described later. The arm unit 11 corresponds to the arm unit 5031 shown in FIG. 1. In FIG. 4, one joint unit 111 is shown as a representative of the plurality of joints. Specifically, the joint unit 111 rotatably connects the links in the arm unit 11, and drives the arm unit 11 by controlling the rotational drive of the joint unit 11 under the control of the arm control unit 23. In this embodiment, information about the position and orientation of the arm unit 11 can be obtained based on the joint angles and link lengths of the joints 5033, links 5035, and the like included in the arm unit 11. Note that in this embodiment, the arm unit 11 may have motion sensors (not shown) including an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like, in order to obtain information about the position and orientation of the arm unit 11.

[0068] (Image capture unit 12) The imaging unit (medical observation device) 12 is provided at the tip of the arm unit (medical arm) 11, and captures images of various imaging targets. That is, the arm unit 11 supports the imaging unit 12. As described above, the imaging unit 12 may be, for example, an oblique viewer 4100, a forward-viewing endoscope with a wide-angle / cutting function (not shown), an endoscope with a tip bending function (not shown), an endoscope with a multi-directional simultaneous imaging function (not shown), or a microscope, and is not particularly limited.

[0069] Furthermore, the imaging unit 12 captures, for example, an image of the surgical field including various medical instruments, organs, and the like in the patient's abdominal cavity. Specifically, the imaging unit 12 is a camera or the like that can capture images of the subject in the form of video or still images. More specifically, the imaging unit 12 is a wide-angle camera configured with a wide-angle optical system. For example, while the angle of view of a typical endoscope is approximately 80°, the angle of view of the imaging unit 12 according to this embodiment may be 140°. Note that the angle of view of the imaging unit 12 may be less than 140° or may be equal to or greater than 140° as long as it exceeds 80°. The imaging unit 12 transmits an electrical signal (pixel signal) corresponding to the captured image to the control unit 20. The arm unit 11 may also support a medical instrument such as forceps 5023.

[0070] Furthermore, in the embodiment of the present disclosure, the imaging unit 12 may be a stereo endoscope (stereo endoscope) capable of distance measurement. Alternatively, in this embodiment, a depth sensor (distance measuring device) (not shown) may be provided separately from the imaging unit 12. In this case, the imaging unit 12 may be a monocular endoscope. Specifically, the depth sensor may be, for example, a sensor that measures distance using a ToF (Time of Flight) method that measures distance using the return time of pulsed light reflected from a subject, or a structured light method that measures distance using distortion of the pattern by irradiating a grid-shaped pattern light. Alternatively, in this embodiment, the imaging unit 12 itself may be provided with a depth sensor. In this case, the imaging unit 12 can perform distance measurement using the ToF method simultaneously with image capture. Specifically, the imaging unit 12 includes multiple light receiving elements (not shown) and can generate an image and calculate distance information based on pixel signals obtained from the light receiving elements.

[0071] (Light source part 13) The light source unit 13 irradiates light onto an object to be imaged by the imaging unit 12. The light source unit 13 can be realized, for example, by an LED (Light Emitting Diode) for a wide-angle lens. The light source unit 13 may be configured, for example, by combining a normal LED with a lens to diffuse the light. The light source unit 13 may also be configured to diffuse (widen the angle of light) the light transmitted by an optical fiber (light guide) using a lens. The light source unit 13 may also widen the illumination range by irradiating light in multiple directions using the optical fiber itself.

[0072] (Control unit 20) The control unit 20 is realized by, for example, a central processing unit (CPU) or a micro processing unit (MPU) executing a program (for example, a program according to an embodiment of the present disclosure) stored in a storage unit 60 (described later) using a random access memory (RAM) or the like as a work area. The control unit 20 is a controller, and may be realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Specifically, the control unit 20 mainly includes an image processing unit 21, an imaging control unit 22, an arm control unit 23, a reception unit 25, and a display control unit 26.

[0073] The image processing unit 21 performs various processes on the imaged object captured by the imaging unit 12. In particular, the image processing unit 21 acquires the image of the imaged object captured by the imaging unit 12, and generates various images based on the image captured by the imaging unit 12. Specifically, the image processing unit 21 can generate images by cutting out and enlarging a display target area from the image captured by the imaging unit 12. In this case, the image processing unit 21 may change the cut-out position depending on the position and posture of the arm unit 11, for example.

[0074] The imaging control unit 22 controls the imaging unit 12. For example, the imaging control unit 22 controls the imaging unit 12 to capture an image of the surgical field. For example, the imaging control unit 22 controls the magnification ratio of the imaging unit 12. Furthermore, the imaging control unit 22 may control the magnification ratio of the imaging unit 12 based on input information from the surgeon 5067 received by the reception unit 25, for example.

[0075] The imaging control unit 22 also controls the light source unit 13. For example, the imaging control unit 22 controls the brightness of the light source unit 13 when the imaging unit 12 images the surgical field. The imaging control unit 22 controls the brightness of the light source unit 13 based on input information from the surgeon 5067 received by the reception unit 25, for example.

[0076] The arm control unit 23 comprehensively controls the robot arm device 10 and also controls the drive of the arm unit 11. Specifically, the arm control unit 23 controls the drive of the joint unit 11a, thereby controlling the drive of the arm unit 11. More specifically, the arm control unit 23 controls the amount of current supplied to the motor in the actuator of the joint unit 11a, thereby controlling the number of rotations of the motor, and thereby controlling the rotation angle and generated torque of the joint unit 11a.

[0077] The arm control unit 23 can autonomously control the position and posture of the arm unit 11, for example, based on a learning model described below. The arm control unit 23 autonomously controls the position and posture of the arm unit 11 using a learning model (trained model) obtained by performing machine learning using, for example, input information from the surgeon 5067 received by the reception unit 25 and various data (external input information) such as images obtained by the imaging unit 12 as learning data. In this case, the arm control unit 23 may control the position and posture of the arm unit 11 by driving the joint 11a of the arm unit 11, for example, so as to avoid a medical instrument that obstructs the field of view of the surgeon 5067. The arm control unit 23 can autonomously control the position and posture of the arm unit 11, for example, based on a learning model described below. The arm control unit 23 autonomously controls the position and posture of the arm unit 11 by referring to a learning model obtained by machine learning learning data based on, for example, input information from the surgeon 5067 received by the reception unit 25 and various data (external input information) such as images obtained by the imaging unit 12. In this case, the arm control unit 23 may control the position and posture of the arm unit 11 by driving the joint 11a of the arm unit 11, for example, so as to avoid medical instruments that obstruct the field of view of the surgeon 5067.

[0078] The learning model may be generated based on, for example, data related to other surgeries. The data related to the surgeries may include, for example, information related to medical instruments used in the surgeries. Here, the information related to the medical instruments may include, for example, image data generated by the medical instruments and information related to the operation of the medical instruments. In an embodiment of the present disclosure, the accuracy of discrimination can be improved by using a learning model generated based on image data captured by various medical instruments and information related to the operation of various medical instruments.

[0079] Specifically, the training data may include sensing data obtained from at least one of a stereo sensor, a depth sensor, and a motion sensor, for example. More specifically, the training data may include information on the surgical field environment, including at least one of the position, posture, type, and movement of a medical instrument and the position, posture, and type of an organ, obtained from at least one of a stereo sensor, a depth sensor, and a motion sensor.

[0080] Furthermore, data related to surgery used when generating a trained model may include information related to the arm unit 11. The information related to the arm unit 11 may include, for example, information related to the state of the joint 11a of the arm unit 11. The information related to the state of the joint 11a of the arm unit 11 may include, for example, various information such as the position, posture, and movement of the joint of the arm unit 11. In an embodiment of the present disclosure, by using a trained model generated based on various information related to the arm unit 11, it is possible to improve the accuracy of discrimination.

[0081] Furthermore, the information about the arm unit 11 may include information about a medical instrument held by the arm unit 11. The information about the medical instrument may include, for example, at least one of the type of the medical instrument, and position information and posture information about the medical instrument.

[0082] In addition, in the embodiment of the present disclosure, the control unit 20 may have a function of generating a learning model. In this case, the control unit 20 generates the learning model and stores the generated learning model in the storage unit 60 described later.

[0083] The learning model used in the embodiments of the present disclosure is generated by learning features of various types of input information used when classifying input information and performing processing according to the classification results. The learning model may be realized by a deep neural network (DNN), which is a multi-layer neural network having multiple nodes including an input layer, multiple intermediate layers (hidden layers), and an output layer. However, the embodiments of the present disclosure are not limited to this. For example, to generate a learning model, first, various types of input information are input via the input layer, and multiple intermediate layers connected in series perform processing such as extraction of features of the input information. Next, various processing results, such as classification results based on the information output by the intermediate layers, are output via the output layer as output information corresponding to the input information, thereby generating a learning model.

[0084] The control unit 20 may generate learning models for various surgeries or may store predetermined models. The control unit 20 generates the learning models from learning data including, for example, a procedure status including information about the procedure performed by the surgeon and endoscopic examination data related to camera operation by the scopist. The control unit 20 generates the learning models using, for example, the positions and postures of medical instruments and endoscopes measured by a tracking device as learning data. The control unit 20 generates the learning models using, for example, endoscopic images including the depth and movement of each object captured by a stereo endoscope and the type of medical instrument as learning data.

[0085] The control unit 20 may generate a learning model using data related to various surgical robots as learning data. The control unit 20 may generate a learning model using, for example, various operation information on the surgical robot by doctors and scopists as learning data. The control unit 20 may generate a learning model using, for example, a procedure using the support arm device 5027 shown in FIG. 1 as learning data.

[0086] The reception unit 25 receives input operations by the surgeon 5067 and various types of input information from other devices (sensors), and outputs the information to the imaging control unit 22 and the arm control unit 23. Details of the input information will be described later.

[0087] The display control unit 26 causes various images to be displayed on the presentation device 40 (described later). The display control unit 26 causes the presentation device 40 to display an image acquired from the imaging unit 12, for example.

[0088] (Presentation device 40) The presentation device 40 displays various images. For example, the presentation device 40 displays an image captured by the imaging unit 12. The presentation device 40 can be a display including, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

[0089] (Storage unit 60) The storage unit 60 stores various types of information. For example, the storage unit 60 stores a learning model. The storage unit 60 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0090] <<3. Background leading to the creation of the embodiments of the present disclosure>> In recent years, developments have been underway to autonomously operate the robot arm device 10 in the above-mentioned medical observation system 1. For example, the autonomous operation of the robot arm device 10 in the medical observation system 1 can be divided into various levels. Examples include (1) a level where the system guides the surgeon (doctor) 5067, and (2) a level where the system autonomously performs some of the operations (tasks) in surgery, such as moving the position of the imaging unit 12 and suturing the surgical site, which are normally performed by the scopist. Furthermore, examples include (3) a level where the system automatically generates the operation content for surgery, and the robot arm device 10 performs the operation selected by the doctor from the automatically generated operations. Furthermore, in the future, (4, 5) levels where the robot arm device 10 performs all of the tasks in surgery, with or without the supervision of a doctor, are also conceivable.

[0091] In the embodiment of the present disclosure described below, an example will be described in which the robot arm device 10 autonomously executes the task of moving the position of the imaging unit 12 instead of the scopist, and the surgeon 5067 performs surgery directly or remotely by referring to the image captured by the moved imaging unit 12.

[0092] For the robot arm device 10 to operate autonomously, it is necessary to generate autonomous operation control information (e.g., autonomous control target values, etc.) for the autonomous operation in advance. Therefore, autonomous operation control information based on actual internal body environment information (e.g., the state of the affected area, etc.) is generated by referring to a learning model and operation rules obtained in advance by machine learning internal body environment information (e.g., the three-dimensional structure of the abdominal cavity, organ state, and affected area state) and the corresponding surgical actions of the surgeon 5067. However, since each patient has different body shapes, organ morphologies, organ positions, etc., it is practically difficult to machine learn all internal body environment information in advance, and it is inevitable that unknown internal body environments exist that cannot be guaranteed by the learning model, etc. Therefore, for example, in a surgery performed in an environment where the imaging unit 12 (e.g., the endoscope 5001) moves in accordance with autonomous operation control information, if the internal body environment is not guaranteed by the autonomous operation control information, there is a concern that the field of view of the imaging unit 12 may be shifted in response to forceps manipulation, or that the imaging unit 12 may interfere with an object other than the target (an organ or tissue) (the field of view of the imaging unit 12 may be blocked by an object other than the target, or the imaging unit 12 itself may collide with an object), making it difficult to perform the surgery suitably and smoothly.

[0093] Furthermore, for unknown internal body environments, it is conceivable to reconstruct a learning model after acquiring information about the internal body environment, and then generate autonomous operation control information using the reconstructed learning model. However, because reconstruction requires a certain amount of preparation time, it is difficult to generate a learning model or autonomous operation control information in real time. Furthermore, even if it were possible to generate a learning model or autonomous operation control information in real time, there would be a delay in the movement of the imaging unit 12 relative to the forceps manipulation by the surgeon 5067. Therefore, in order to deal with such delays, it is conceivable that the surgeon 5067 would manipulate the forceps while predicting changes in the image of the imaging unit 12. However, because this involves difficult predictions, it is still difficult to perform surgery appropriately and smoothly.

[0094] In view of the above, the present inventors have created an embodiment of the present disclosure that is capable of generating autonomous movement control information for a robot arm device 10 in real time, suitable for various intracorporeal environments. In the embodiment of the present disclosure, autonomous movement control information generated based on various data (external input information) is modified based on the results of a motion simulation of the robot arm device 10 in an actual intracorporeal environment and the results of a motion simulation of the robot arm device 10 in an ideal intracorporeal environment, thereby enabling the generation of autonomous movement control information suitable for various intracorporeal environments in real time. The surgeon 5067 can then perform surgery smoothly and efficiently by receiving assistance from the robot arm device 10 controlled in accordance with such autonomous movement control information. The following describes the details of the embodiment of the present disclosure created by the present inventors.

[0095] <<4. Embodiment>> The embodiment of the present disclosure created by the inventors can be divided into two main stages: a stage of generating in advance a learning model, etc. to be used when generating autonomous operation control information (learning model generation stage), and a stage of performing autonomous operation based on the generated learning model, etc. (autonomous operation execution stage). First, the learning model generation stage according to this embodiment will be described.

[0096] <4.1 Detailed configuration example of the learning model generation unit 100> First, with reference to FIG. 5, a detailed configuration example of the learning model generation unit 100 according to an embodiment of the present disclosure will be described. FIG. 5 is a block diagram showing an example of the configuration of the learning model generation unit 100 according to an embodiment of the present disclosure. The learning model generation unit 100 according to this embodiment can generate a learning model 132, a rule-based control model 134, and an ideal internal body environment map 136 used when generating autonomous movement control information (specifically, autonomous movement target values). Note that the learning model generation unit 100 may be a device integrated with the robot arm device 10 or the control unit 20 shown in FIG. 4 described above, or may be a separate device. Alternatively, it may be a device provided on the cloud and communicably connected to the robot arm device 10 and the control unit 20.

[0097] 5, the learning model generation unit 100 mainly includes an external input information acquisition unit 110, an autonomous movement learning model generation unit 120, an autonomous movement rule generation unit 122, an ideal map generation unit 124, a storage unit 130, and an output unit 140. Details of each functional unit of the learning model generation unit 100 will be explained below in order.

[0098] (External input information acquisition unit 110) The external input information acquisition unit 110 acquires various data (external input information) and outputs it to the autonomous movement learning model generation unit 120, the autonomous movement rule generation unit 122, and the ideal map generation unit 124. In this embodiment, the various data may include, for example, tip position and orientation data of each forceps (not shown) held by both hands in an ideally performed surgery (position information and orientation information of the medical instrument in the real environment), tip position and orientation data of the imaging unit 12 (more specifically, the endoscope 5001) (position information and orientation information of the medical arm in the real environment), an image of the imaging unit 12 (an image of the real environment captured by a medical observation device), and sensing data from a distance measurement sensor (such as a stereo endoscope or a depth sensor) (not shown). Specifically, in this embodiment, the tip position and orientation data may be acquired by using, for example, a motion sensor (not shown) provided in the arm unit 11 or sensing data from the above-mentioned distance measurement sensor (a stereo endoscope (stereo camera), a structured light sensor, or a ToF sensor). In this embodiment, the information on the tip position and orientation data may be expressed as absolute coordinate values ​​or relative coordinate values ​​from a predetermined reference point, and is not particularly limited.

[0099] (Autonomous action learning model generation unit 120) The autonomous movement learning model generation unit 120 performs machine learning using, for example, pairs of tip position / orientation data of each forceps (not shown) held by both hands and the imaging unit 12 output from the external input information acquisition unit 110 as training data, and generates a learning model 132 that outputs the position / orientation of the imaging unit 12 with respect to the position / orientation of each forceps held by any one of the left and right hands. In this embodiment, by performing machine learning using a large number of pairs of tip position / orientation data of each forceps held by both hands and the imaging unit 12 in an ideally performed surgery, it is possible to generate a learning model 132 for realizing autonomous movement of the arm unit 11 of the robot arm device 10, etc.

[0100] Alternatively, the autonomous movement learning model generation unit 120 can perform machine learning using a pair of an image of the imaging unit 12 and the tip position / orientation data of the imaging unit 12 as training data, and generate a learning model 132 that outputs visual field information consisting of information such as the position of a gaze point located at the center of the visual field of the imaging unit 12, the distance from the imaging unit 12 to the gaze point, and the visual field direction to the gaze point, for an arbitrary tip position / orientation of the imaging unit 12. The learning model 132 generated by the autonomous movement learning model generation unit 120 is stored in the storage unit 130, which will be described later.

[0101] (Autonomous operation rule generation unit 122) The autonomous operation rule generation unit 122 analyzes a plurality of data collected during an ideally performed surgery, thereby generating a rule-based control model 134 that defines control rules (such as the allowable distance and positional relationship with an organ, the allowable distance and positional relationship with forceps, etc.) that are permitted during the autonomous operation of the arm unit 11, etc. of the robot arm device 10. The rule-based control model 134 generated by the autonomous operation rule generation unit 122 is stored in the storage unit 130, which will be described later.

[0102] Specifically, the autonomous operation rule generation unit 122 generates the rule-based control model 134 by extracting rules for the position and orientation of the image capture unit 12 with respect to the position and orientation of each of the forceps (not shown) held by any of the left and right hands from pairs of tip position and orientation data of the image capture unit 12 and each of the forceps (not shown) held by any of the left and right hands output from the external input information acquisition unit 110. Alternatively, the autonomous operation rule generation unit 122 generates the rule-based control model 134 by extracting rules for the visual field information of the image capture unit 12 with respect to any image of the image capture unit 12 from pairs of the image of the image capture unit 12 and the tip position and orientation data of the image capture unit 12.

[0103] (Ideal map generation unit 124) The ideal map generating unit 124 generates multiple internal body environment maps based on various data used in generating the learning model 132 and the rule-based control model 134. Here, the 3D map information (internal body environment map) within the body that realizes the most ideal operation (also referred to as scope work) of the imaging unit 12 (specifically, the arm unit 11) is referred to as the "ideal internal body environment map." The internal body environment that realizes the most ideal scope work is referred to as the ideal environment or ideal internal body environment. The ideal internal body environment map 136 generated by the ideal map generating unit 124 is stored in the storage unit 130, which will be described later, and is used during the simulation, which will be described later. Note that in this embodiment, the ideal internal body environment map 136 does not need to be limited to one, and multiple maps may exist. For example, the simulation execution environment may be updated for each scene, and one of the multiple ideal internal body environment maps 136 may be selected depending on the function of the updated simulator. In this specification, "ideal scope work" refers to scope work that allows the surgeon 5067 to perform a procedure without removing the operative site (the area of ​​the organ, tissue, etc. being treated) and the forceps operating section (the area located at the tip of the forceps, where tissue is grasped, etc.) from the field of view, and that follows the movement of the forceps tips when inserting and removing the forceps, capturing them in the center of the screen, while providing images displayed at an appropriate magnification so that the surgeon 5067 can visually recognize the surrounding organs, tissue, and the tip of the forceps. In other words, in this specification, "ideal scope work" can be rephrased as "appropriate scope work" that can provide the above-mentioned images. In this specification, "forceps operation" may include not only resection and grasping operations by opening and closing the tip of the forceps, but also movement of the tip of the forceps, such as changing the position of the tip of the forceps and inserting and removing the forceps. In addition, in this specification, an "ideal intracorporeal environment map" is information on one map pattern that allows a learning model to realize ideal scope work. In other words, it does not simply mean 3D map information of the inside of the body, but a known 3D map of the inside of the body that can achieve near-ideal scope work using learning models and rule-based control models stored in a database.

[0104] Specifically, the ideal map generating unit 124 generates the ideal intracorporeal environment map 136 using, for example, SLAM (Simultaneous Localization and Mapping) from position information of organs extracted from the image and the position and orientation information of the image capturing unit 12, based on a pair of the image of the image capturing unit 12 and the position and orientation data of the tip of the image capturing unit 12. Alternatively, the ideal map generating unit 124 generates the ideal intracorporeal environment map 136 using SLAM from position information of organs extracted from the image and the position and orientation information of the image capturing unit 12, based on a pair of the image of the image capturing unit 12 and sensing data from a distance measuring sensor (stereo endoscope, depth sensor, etc.) (not shown). Note that the ideal intracorporeal environment map 136 is preferably a metric map using a representation method such as grid, boxel, or point cloud.

[0105] (Storage unit 130) The memory unit 130 stores the above-mentioned learning model 132, rule-based control model 134, and ideal internal body environment map 136. The memory unit 130 is realized by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disk.

[0106] (output unit 140) The output unit 140 can output the learning model 132, the rule-based control model 134, and the ideal internal body environment map 136 to the autonomous action execution unit 200, which will be described later.

[0107] In this embodiment, the detailed configuration of the learning model generation unit 100 is not limited to the configuration shown in FIG.

[0108] 4.2 Control method at the learning model generation stage Next, a control method at the learning model generation stage according to this embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart of the control method at the learning model generation stage according to this embodiment. As shown in Fig. 8, the control method at the learning model generation stage according to this embodiment can mainly include steps S10 to S40. Below, an overview of each of these steps according to this embodiment will be described.

[0109] First, the external input information acquisition unit 110 of the learning model generation unit 100 acquires various data (external input information) collected during an ideally performed surgery (step S10). Next, the autonomous movement learning model generation unit 120 performs machine learning using the various data as training data to generate a learning model 132 and outputs it to the storage unit 130 and the autonomous movement execution unit 200 (step S20). Furthermore, the autonomous movement rule generation unit 122 analyzes the various data (external input information) to generate a rule-based control model 134 and outputs it to the storage unit 130 and the autonomous movement execution unit 200 (step S30). Then, the ideal map generation unit 124 generates an internal body environment map based on the various data and outputs it to the storage unit 130 and the autonomous movement execution unit 200 (step S40).

[0110] 4.3 Detailed configuration example of the autonomous operation execution unit 200 Next, a stage (autonomous operation execution stage) of performing autonomous operation according to this embodiment will be described. A detailed configuration example of the autonomous operation execution unit 200 according to this embodiment of the present disclosure will be described with reference to FIG. 7 . FIG. 7 is a block diagram showing an example of the configuration of the autonomous operation execution unit 200 according to this embodiment. The autonomous operation execution unit 200 according to this embodiment can control the robot arm device 10 (e.g., the arm section 11) so as to perform autonomous operation in real time based on various data (external input information). In detail, as shown in FIG. 7 , the autonomous operation execution unit 200 is mainly composed of two units: a control unit 210 and a simulator unit 250. Note that the autonomous operation execution unit 200 may be a device integrated with the robot arm device 10 or the control unit 20 shown in FIG. 4 described above, a separate device, or a device provided on the cloud and communicably connected to the robot arm device 10 and the control unit 20.

[0111] ~Control Unit 210~ First, the control unit 210 will be described. In detail, as shown in Fig. 7, the control unit 210 mainly includes an input information acquisition unit 212, an autonomous action generation unit (control information generation unit) 214, an autonomous action correction unit (correction unit) 216, a camera predicted view calculation unit 218, a view interlocking control calculation unit (avoidance control information generation unit) 220, a user instruction arbitration unit 224, an arm control unit (control unit) 226, a tip control unit 228, and a cut-out view generation unit (predicted image generation unit) 230. Details of each functional unit of the control unit 210 will be sequentially described below.

[0112] (Input information acquisition unit 212) The input information acquisition unit 212 acquires various data (external input information) obtained during an actual surgery or the like currently being performed (in a real environment) and outputs the data to the autonomous operation generation unit 214 and the simulator unit 250, which will be described later. In this embodiment, the various data may include, for example, tip position and orientation data of each forceps (not shown) held by both hands (position information and orientation information of the medical instrument in the real environment), tip position and orientation data of the imaging unit 12 (position information and orientation information of the medical arm in the real environment), an image of the imaging unit 12 (an image of the real environment captured by a medical observation device), and sensing data from a distance sensor (such as a stereo endoscope or a depth sensor) (not shown). In this embodiment, the tip position and orientation data may be acquired in real time by using sensing data from a motion sensor (not shown) provided on the arm unit 11 or the distance sensor described above. In addition, in this embodiment, when the device is operated in a master-slave mode, the tip position and orientation data may be generated using information input to the input device 5047 (master console). In this embodiment, the information on the tip position and orientation data may be expressed as absolute coordinate values ​​or relative coordinate values ​​from a predetermined reference point, and is not particularly limited.

[0113] (Autonomous motion generation unit 214) The autonomous action generation unit 214 can generate autonomous action target values ​​(autonomous action control information) for autonomously operating the robot arm device 10 (e.g., the arm unit 11) (medical arm) based on various data (external input information) output from the input information acquisition unit 212. Specifically, the autonomous action generation unit 214 references the learning model 132 and the rule-based control model 134 described above, and generates, as the autonomous action target values, information on the position and orientation of the image capture unit 12 (e.g., the position and orientation of the image capture unit 12 for capturing only the forceps within the field of view) based on actual tip position and orientation data of each forceps (not shown) held in both hands. Alternatively, the autonomous action generation unit 214 generates, as the autonomous action target values, visual field information including information such as the position of a gaze point located at the center of the visual field of the image capture unit 12, the distance from the image capture unit 12 to the gaze point, and the visual field direction to the gaze point. The autonomous action generation unit 214 then outputs the generated autonomous action target values ​​to the autonomous action correction unit 216, which will be described later.

[0114] (Autonomous operation modification unit 216) The autonomous action correction unit 216 can correct the autonomous action target value (autonomous action control information) in real time based on the result of the action simulation of the robot arm device 10 (e.g., the arm unit 11). In particular, the autonomous action correction unit 216 can correct the autonomous action target value (autonomous action control information) in real time based on the result of the action simulation of the robot arm device 10 (e.g., the arm unit 11) based on the actual current internal body environment (real environment) and the result of the action simulation of the robot arm device 10 (e.g., the arm unit 11) based on an ideal internal body environment (ideal environment). More specifically, the autonomous action correction unit 216 corrects the autonomous action target value generated by the autonomous action generation unit 214 in real time based on an autonomous action correction parameter (difference) acquired from the simulator unit 250 (described later). The autonomous action correction unit 216 then outputs the corrected autonomous action target value to the visual field linkage control calculation unit 220 (described later).

[0115] (Camera predicted field of view calculation unit 218) The camera predicted view calculation unit 218 determines a target view of the imaging unit 12 for presenting an interfering object in advance and for collision avoidance based on interference prediction information (such as the position of an organ predicted to interfere, and position and posture information of the imaging unit 12 at the time of interference) acquired from a simulator unit 250 (described later). In particular, if the imaging unit 12 is an endoscope (not shown) with a tip bending function, the camera predicted view calculation unit 218 outputs information on the target posture and target view of the tip of the imaging unit 12 to a view interlocking control calculation unit 220 (described later). Furthermore, if the imaging unit 12 is an endoscope (not shown) with a wide-angle / cropping function, the camera predicted view calculation unit 218 outputs information on the view range and view to be cropped to a cropped view generation unit 230 (described later).

[0116] (Visual field interlocking control calculation unit 220) The visual field interlocking control calculation unit 220 can generate control information (tip degree of freedom control command value, arm control command value) for operating the robot arm device 10 (e.g., the arm unit 11) while avoiding interference, based on the corrected autonomous operation target value and the target posture (interference prediction information) of the tip of the imaging unit 12. That is, in this embodiment, if the imaging unit 12 is an endoscope (not shown) with a tip bending function, the visual field interlocking control calculation unit 220 can generate control information for controlling the arm unit 11 and the tip of the imaging unit 12 to move in a coordinated manner. Then, the visual field interlocking control calculation unit 220 outputs the generated control information to a user instruction arbitration unit 224 (described later). Note that if the imaging unit 12 is not an endoscope (not shown) with a tip bending function, the visual field interlocking control calculation unit 220 outputs the corrected autonomous operation target value to the user instruction arbitration unit 224 as is.

[0117] (User instruction arbitration unit 224) The user instruction arbitration unit 224 can update the control information (tip control command value, arm control command value) based on the control information (control command value) output from the visual field interlocking control calculation unit 220 and a correction command (for example, a gaze point offset or a zoom amount offset) manually input by the surgeon (user) 5067. Then, the user instruction arbitration unit 224 outputs the updated control information (tip control command value, arm control command value) to the arm control unit 226 and the tip control unit 228 described below. Note that in this embodiment, if there is no input from the surgeon (user) 5067, the control information (control command value) output from the visual field interlocking control calculation unit 220 is output as is to the arm control unit 226 and the tip control unit 228.

[0118] (Arm control unit 226) The arm control unit 226 can control the arm unit (medical arm) 11 based on the updated control information (the arm control command value).

[0119] (Front end control unit 228) The tip control unit 228 can control the direction (field of view) of the tip of the imaging unit 12, the zoom amount, the attitude of the tip of the imaging unit 12, etc., in synchronization with the above-mentioned arm control unit 226 based on the updated control information (tip control command value).

[0120] (Cutting field of view generation unit 230) The crop field of view generation unit 230 generates a predicted image that would be cropped when cropping processing is performed on the wide-angle image acquired by the imaging unit 12, based on the cropping field of view range and field of view acquired from the camera predicted field of view calculation unit 218, and outputs the predicted image to the presentation device 40.

[0121] ~Simulator Unit 250~ Next, the simulator unit 250 will be described. The simulator unit 250 can perform a motion simulation of the robot arm device 10 (e.g., the arm section 11) based on various data (external input information). In detail, as shown in FIG. 7 , the simulator unit 250 mainly includes an input information receiving unit 252, an ideal environment map holding unit 254, an internal body environment map generating unit 256, an ideal environment input correcting unit 258, an ideal environment autonomous action generating unit 260, an ideal environment simulation unit 262, a current environment autonomous action generating unit 264, a current environment simulation unit (real environment simulation unit) 266, an autonomous action result comparing unit (comparison unit) 268, an autonomous action correction parameter generating unit 270, an autonomous action correcting unit 272, a current environment simulation updating unit (re-simulation unit) 274, an interference predicting unit (prediction unit) 276, and an image generating unit 278. The following describes the details of each functional unit of the simulator unit 250 in order.

[0122] (Input information receiving unit 252) The input information receiving unit 252 outputs data used for generating an environmental map from the input information acquiring unit 212 of the control unit 210 described above to the internal environment map generating unit 256, and outputs data used for generating data for autonomous operation to the ideal environment input correcting unit 258 and the current environment autonomous operation generating unit 264. Furthermore, the input information receiving unit 252 outputs to the internal environment map generating unit 256 the tip position and orientation data of each forceps (not shown) held in both hands, the tip position and orientation data of the imaging unit 12, images of the imaging unit 12, sensing data from a distance measuring sensor (not shown), etc.

[0123] (Ideal environment map storage unit 254) The ideal environment map storage unit 254 stores the ideal internal body environment map 136 generated by the ideal map generation unit 124 , and outputs it to the ideal environment input correction unit 258 and the ideal environment simulation unit 262 .

[0124] (Internal environment map generation unit 256) The internal body environment map generating unit 256 can generate an actual current internal body environment map (real environment map) in real time based on various data (external input information) acquired from the input information receiving unit 252, including tip position and orientation data of each forceps (not shown) held in both hands, tip position and orientation data of the imaging unit 12, images of the imaging unit 12, sensing data from a distance measuring sensor (not shown), etc. For example, the internal body environment map generating unit 256 generates a current internal body environment map using the SLAM method based on images from the imaging unit 12. The generated current internal body environment map is then used by the ideal environment input correcting unit 258, described later, to correct data by comparing it with the ideal internal body environment map. Here, the 3D map information (internal body environment map) of the inside of the body currently undergoing surgery (referred to as the current environment (real environment) or current internal body environment) is referred to as the "current internal body environment map."

[0125] (Ideal environment input correction unit 258) The ideal environment input correction unit 258 extracts feature points between the ideal internal body environment map information stored in the ideal environment map storage unit 254 and the current internal body environment map generated by the internal body environment map generation unit 256, for example, using the current tip position and orientation data of the imaging unit 12 and the tip position and orientation data of each forceps (not shown) held in both hands, and calculates the difference between the extracted feature points. Then, based on the calculated difference, the ideal environment input correction unit 258 can correct data (external input information) used to generate data for autonomous operation, acquired from the input information receiving unit 252. For example, the ideal environment input correction unit 258 outputs various data (external input information), such as the tip position and orientation data of each forceps held in both hands in the corrected ideal environment, to the ideal environment autonomous operation generation unit 260.

[0126] (Ideal environment autonomous movement generation unit 260) The ideal environment autonomous movement generation unit 260 can generate autonomous movement target values ​​(autonomous movement control information in an ideal environment) for the robot arm device 10 (e.g., the arm unit 11) based on various corrected data (external input information) with reference to the learning model 132 or the rule-based control model 134. In detail, the ideal environment autonomous movement generation unit 260 generates, as autonomous movement target values, information on the position and orientation of the image capture unit 12 (e.g., the position and orientation of the image capture unit 12 for capturing only each forceps within the field of view) based on tip position and orientation data of each forceps (not shown) held in both hands with reference to the above-mentioned learning model 132 and rule-based control model 134. Alternatively, the ideal environment autonomous movement generation unit 260 generates, as autonomous movement target values, visual field information including information on the position of a gaze point located at the center of the visual field of the image capture unit 12, the distance from the image capture unit 12 to the gaze point, the visual field direction to the gaze point, etc. The ideal environment autonomous movement generation unit 260 then outputs the generated autonomous movement target values ​​to the ideal environment simulation unit 262.

[0127] (Ideal Environment Simulation Department 262) The ideal environment simulation unit 262 can perform a movement simulation of the robot arm device 10 (for example, the arm unit 11) in the ideal environment, based on the ideal intracorporeal environment map (ideal environment map) stored in the ideal environment map storage unit 254 and the autonomous movement target values ​​generated by the ideal environment autonomous movement generation unit 260. The ideal environment simulation unit 262 outputs, as simulation results, to the autonomous movement result comparison unit 268, an image of the imaging unit 12 in the ideal intracorporeal environment, positional relationship information between the imaging unit 12 and the forceps, and positional relationship information between the imaging unit 12 and surrounding organs.

[0128] (Currently environmental autonomous movement generation unit 264) The current environment autonomous movement generation unit 264 can generate autonomous movement target values ​​(autonomous movement control information in the real environment) for the robot arm device 10 (e.g., the arm unit 11) based on various data (external input information) from the input information receiving unit 252, with reference to the learning model 132 and the rule-based control model 134. Specifically, the current environment autonomous movement generation unit 264 generates, as the autonomous movement target values, information on the position and orientation of the image capture unit 12 (e.g., the position and orientation of the image capture unit 12 for capturing only the forceps within the field of view) based on the tip position and orientation data of each forceps (not shown) held in both hands, with reference to the learning model 132 and the rule-based control model 134. Alternatively, the current environment autonomous movement generation unit 264 generates, as the autonomous movement target values, visual field information including information on the position of a gaze point located at the center of the visual field of the image capture unit 12, the distance from the image capture unit 12 to the gaze point, the visual field direction to the gaze point, etc. The current environment autonomous movement generation unit 264 then outputs the generated autonomous movement target values ​​to the current environment simulation unit 266, which will be described later.

[0129] (Currently Environmental Simulation Department 266) The current environment simulation unit 266 can perform a motion simulation of the robot arm device 10 (e.g., the arm unit 11) in the current environment (real environment) based on the current internal body environment map (real environment map) generated in real time by the internal body environment map generation unit 256 and the autonomous movement target values ​​generated by the current environment autonomous movement generation unit 264. The current environment simulation unit 266 outputs, as simulation results, to the autonomous movement result comparison unit 268, an image of the imaging unit 12 in the current internal body environment, positional relationship information between the imaging unit 12 and the forceps, and positional relationship information between the imaging unit 12 and surrounding organs.

[0130] (Autonomous operation result comparison unit 268) The autonomous movement result comparison unit 268 can extract the difference between the result of the movement simulation in the ideal environment by the ideal environment simulation unit 262 and the result of the movement simulation in the current environment (real environment) by the current environment simulation unit 266. In detail, based on the image of the imaging unit 12 in the ideal environment, the image of the imaging unit 12 in the current environment, positional relationship information between the imaging unit and the forceps, positional relationship information between the imaging unit 12 and the surrounding organs in the ideal environment and the current environment, the autonomous movement result comparison unit 268 outputs the difference in feature points of the image of the imaging unit 12, the difference in the positional relationship between the imaging unit 12 and the forceps, and the difference in the positional relationship between the imaging unit 12 and the surrounding organs to the autonomous movement correction parameter generation unit 270.

[0131] (Autonomous action correction parameter generation unit 270) The autonomous movement correction parameter generation unit 270 generates correction parameters (such as the offset of the position and orientation of the image capture unit 12 and the zoom amount) for correcting the autonomous movement target of the robot arm device 10 (for example, the arm unit 11) based on the difference acquired from the autonomous movement result comparison unit 268. Then, the autonomous movement correction parameter generation unit 270 outputs the generated correction parameters to the autonomous movement correction units 216, 272.

[0132] (Autonomous operation modification unit 272) The autonomous movement correction unit 272 can correct the autonomous movement target values ​​(autonomous movement control information in the real environment) of the robot arm device 10 (for example, the arm unit 11) based on the autonomous movement target values ​​(autonomous movement control information in the real environment) of the robot arm device 10 (for example, the arm unit 11) generated by the current environment autonomous movement generation unit 264 and the correction parameters (offset of the position and orientation of the image capture unit 12 and zoom amount) generated by the autonomous movement correction parameter generation unit 270. Then, the autonomous movement correction unit 272 outputs the corrected autonomous movement target values ​​to the current environment simulation update unit 274, which will be described later.

[0133] (Current Environmental Simulation Update Part 274) The current environment simulation update unit 274 can perform a motion simulation again in the current environment based on the autonomous movement target values ​​corrected by the autonomous movement correction unit 272. In detail, the current environment simulation update unit 274 can perform a motion simulation again of the robot arm device 10 (e.g., the arm unit 11) in the current environment based on the current internal body environment map (real environment map) updated in real time by the internal body environment map generation unit 256 and the autonomous movement target values ​​corrected by the autonomous movement correction unit 272. Then, the current environment simulation update unit 274 outputs, as simulation results, an image of the imaging unit 12 in the current environment, positional relationship information between the imaging unit 12 and the forceps, and positional relationship information between the imaging unit 12 and surrounding organs to an interference prediction unit 276 and an image generation unit 278, which will be described later.

[0134] (Interference prediction unit 276) The interference prediction unit 276 can perform interference prediction and contact determination with surrounding organs (other objects) based on the results of the re-simulation in the current environment simulation update unit 274, and output interference prediction information (information such as the position of the interfering organ (other part) where interference is predicted, and the position and posture of the imaging unit 12 (or arm unit 11) at the time of interference) to the control unit 210.

[0135] (Image generation unit 278) Based on the results of the re-simulation in the current environment simulation update unit 274, the image generation unit 278 can generate a predicted image (interference predicted image) that will be obtained by the imaging unit 12 when it interferes with the interfering organ, and output it to the presentation device 40.

[0136] In this embodiment, the detailed configuration of the autonomous operation execution unit 200 is not limited to the configuration shown in FIG.

[0137] ~Presentation device 40~ Next, the presentation device 40 will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the configuration of the presentation device 40 according to this embodiment. In detail, as shown in Fig. 8, the presentation device 40 mainly includes a real image acquisition unit 402, a virtual image acquisition unit 404, and a prediction result generation unit 406. Details of each functional unit of the presentation device 40 will be sequentially described below.

[0138] (Actual image acquisition unit 402) The actual image acquisition unit 402 can acquire from the control unit 210 an actual image (e.g., a wide-angle image) from the imaging unit 12 and a predicted image that will be cut out, generated by the above-mentioned cut-out field of view generation unit 230, and output them to the prediction result generation unit 406 described later.

[0139] (Virtual image acquisition unit 404) The virtual image acquisition unit 404 can acquire from the simulator unit 250 an interference prediction image that would be obtained by the imaging unit 12 when it interferes with an interfering organ, and output it to the prediction result generation unit 406 described later.

[0140] (Prediction result generation unit 406) The prediction result generation unit 406 can simultaneously present to the surgeon (user) 5067 one, two, or three of the actual image (e.g., a wide-angle image) from the imaging unit 12, the predicted image that will be cut out, and the predicted image predicted from the simulator unit 250 (e.g., an interference predicted image that will be obtained by the imaging unit 12 when it interferes with an interfering organ).

[0141] In this embodiment, the detailed configuration of the presentation device 40 is not limited to the configuration shown in FIG.

[0142] 4.4 Control method during autonomous operation execution ~ Overview ~ Next, a control method at the autonomous operation execution stage according to this embodiment will be described. First, an overview of the control method according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart of the control method at the autonomous operation execution stage according to this embodiment.

[0143] 9, the control method at the autonomous operation execution stage according to this embodiment can mainly include steps S100 to S300. An overview of each of these steps will be described below. The control method described below starts by acquiring various data (external input information), and is repeatedly executed until the surgical task of the autonomous operation execution unit 200 is completed.

[0144] First, the autonomous action execution unit 200 acquires various data (external input information) such as tip position and orientation data of each forceps (not shown) held in both hands (position information and orientation information of the medical instrument in the real environment), tip position and orientation data of the imaging unit 12 (position information and orientation information of the medical arm in the real environment), an image of the imaging unit 12 (an image of the real environment captured by a medical observation device), and sensing data from a distance measurement sensor (stereo endoscope, depth sensor, etc.) (not shown) (step S100).

[0145] Next, the autonomous operation execution unit 200 mainly operates the simulator unit 250 based on the acquired various data to execute a simulation (step S200).

[0146] Furthermore, based on the simulation results of step S200 described above, autonomous operation execution unit 200 mainly operates control unit 210 to control arm section 11 (step S300). Autonomous operation execution unit 200 repeatedly executes the flow shown in Fig. 9 until the task is completed.

[0147] ~Simulator unit operation stage~ Next, details of step S200 shown in FIG. 9 will be described with reference to FIGS. 10 to 20. FIG. 10 is a sub-flowchart of step S200 shown in FIG. 9, and FIGS. 11 to 20 are explanatory diagrams for explaining details of the control method according to this embodiment. As shown in FIG. 10, step S200 may mainly include multiple sub-steps from sub-step S201 to sub-step S215. Details of each of these sub-steps will be described below. Note that in FIG. 10, unlike a normal information processing flowchart, where an arrow branches from one sub-step to multiple sub-steps, this means that multiple sub-steps are processed in parallel.

[0148] The input information receiving unit 252 outputs data used for generating an environmental map from the input information acquiring unit 212 of the control unit 210 described above to the internal environment map generating unit 256, and outputs data used for autonomous operation to the ideal environment input correcting unit 258 and the current environment autonomous operation generating unit 264. Furthermore, the input information receiving unit 252 outputs, for example, tip position and orientation data of each forceps (not shown) held in both hands, tip position and orientation data of the endoscope 5001, an image from the imaging unit 12, sensing data from a distance measuring sensor (not shown), etc. to the internal environment map generating unit 256 (substep S201).

[0149] The internal body environment map generating unit 256 generates a current internal body environment map (real environment map) in real time based on various data (external input information) obtained from the input information receiving unit 252, including tip position and orientation data of each forceps (not shown) held in both hands, tip position and orientation data of the imaging unit 12, images of the imaging unit 12, sensing data from a distance measuring sensor (not shown), etc. (substep S202).

[0150] The autonomous operation generation unit 214 generates an autonomous operation target value (autonomous operation control information) for autonomously operating the robot arm device 10 (e.g., the arm unit 11) based on various data (external input information) output from the input information acquisition unit 212 (substep S203).

[0151] The ideal environment input correcting unit 258 extracts feature points between the ideal internal body environment map information stored in the ideal environment map storage unit 254 and the current internal body environment map generated by the internal body environment map generating unit 256, using, for example, the current tip position and orientation data of the imaging unit 12 and the tip position and orientation data of each forceps (not shown) held in both hands, and calculates the difference between the extracted feature points. For example, as shown in Fig. 11, the ideal environment input correcting unit 258 can extract a difference 500 from the image 136c in the ideal environment by extracting feature points of the outlines of organs in the current internal body image 138.

[0152] Then, based on the difference, ideal environment input correcting unit 258 corrects the input data used for autonomous operation in the ideal environment acquired from input information receiving unit 252. For example, as shown in Fig. 12, ideal environment input correcting unit 258 can calculate a correction amount 502 for correcting the forceps position based on difference 500 in the organ positions. Then, based on the calculated correction amount, ideal environment input correcting unit 258 can correct the forceps position, for example, from the state of image 136a shown on the left side of Fig. 13 to the state of image 136b shown on the right side (substep S204).

[0153] The ideal environment autonomous operation generation unit 260 refers to the learning model 132 and the rule-based control model 134 and generates an autonomous operation target value (autonomous operation control information in the ideal environment) of the robot arm device 10 (e.g., the arm unit 11) in the ideal environment based on the corrected various data (external input information) (substep S205).

[0154] The current environment autonomous movement generation unit 264 generates autonomous movement target values ​​(autonomous movement control information in the real environment) of the robot arm device 10 (e.g., the arm unit 11) in the current environment based on various data (external input information) from the input information receiving unit 252, with reference to the learning model 132 and the rule-based control model 134. Then, the current environment simulation unit 266 performs a movement simulation of the robot arm device 10 (e.g., the arm unit 11) in the current environment based on the current internal body environment map (real environment map) updated in real time by the internal body environment map generation unit 256 and the autonomous movement target values ​​generated by the current environment autonomous movement generation unit 264 (substep S206). For example, the current environment simulation unit 266 simulates an image 602 obtained by the imaging unit 12 in the current environment, as shown in FIG. 14 .

[0155] The ideal environment simulation unit 262 performs a motion simulation of the robot arm device 10 (e.g., the arm unit 11) in the ideal environment (substep S207) based on the ideal internal body environment map information stored in the ideal environment map storage unit 254 and the autonomous movement target values ​​generated by the ideal environment autonomous movement generation unit 260. For example, the ideal environment simulation unit 262 performs a simulation of an image 600 obtained by the imaging unit 12 in the ideal environment, as shown in Fig. 15 (substep S207).

[0156] The autonomous movement result comparison unit 268 extracts a difference between the result of the movement simulation in the ideal environment by the ideal environment simulation unit 262 and the result of the movement simulation in the current environment by the current environment simulation unit 266 (substep S208). For example, as shown in Fig. 16 , the autonomous movement result comparison unit 268 extracts a difference (difference) 504 between an image 600 captured by the imaging unit 12, which is the result of the movement simulation in the ideal environment by the ideal environment simulation unit 262, and an image 602 captured by the imaging unit 12, which is the result of the movement simulation in the current environment by the current environment simulation unit 266.

[0157] The autonomous movement correction parameter generation unit 270 generates correction parameters (offset of the position and orientation of the image capture unit 12 and zoom amount) for correcting the autonomous movement target of the robot arm device 10 (e.g., the arm unit 11) based on the difference acquired from the autonomous movement result comparison unit 268 (substep S209). For example, as shown in Fig. 17 , the autonomous movement correction parameter generation unit 270 calculates the offset amount of the point of interest of the image capture unit 12 and the zoom amount as correction parameters 506.

[0158] The autonomous action execution unit 200 proceeds to the processing of sub-step S302 shown in FIG. 21 (sub-step S210).

[0159] The autonomous movement correction unit 272 corrects the autonomous movement target values ​​of the robot arm device 10 (e.g., the arm unit 11) based on the autonomous movement target values ​​(autonomous movement control information in the real environment) of the robot arm device 10 (e.g., the arm unit 11) generated by the current environment autonomous movement generation unit 264 and the correction parameters (offset of the position and orientation of the image capture unit 12 and zoom amount) generated by the autonomous movement correction parameter generation unit 270 (substep S211). For example, the autonomous movement correction unit 272 corrects the target field of view of the image capture unit 12 based on the correction parameters 506, as shown in FIG. 18 .

[0160] The current environment simulation update unit 274 executes the operation simulation again in the current internal environment based on the autonomous operation target values ​​corrected by the autonomous operation correction unit 272 (substep S212).

[0161] The interference prediction unit 276 performs interference prediction and contact determination with surrounding organs based on the result of the re-simulation by the current environment simulation update unit 274, and outputs interference prediction information to the control unit 210 (substep S213). For example, the interference prediction unit 276 detects interference with surrounding organs in the simulation, as shown in Fig. 19 .

[0162] The image generation unit 278 generates an interference prediction image that will be obtained when interference with the predicted surrounding organs occurs, based on the result of the re-simulation by the current environment simulation update unit 274, and outputs the image to the presentation device 40 (substep S214). For example, the image generation unit 278 generates an interference prediction image 604 as shown in FIG. 20 .

[0163] The autonomous action execution unit 200 proceeds to the processing of sub-step S303 shown in FIG. 21 (sub-step S215).

[0164] ~Control unit operation stage~ Next, details of step S300 shown in FIG. 9 will be described with reference to FIGS. 21 to 25. FIG. 21 is a sub-flowchart of step S300 shown in FIG. 9, and FIGS. 22 to 25 are explanatory diagrams for explaining details of the control method according to this embodiment. As shown in FIG. 21, step S300 can mainly include multiple sub-steps from sub-step S301 to sub-step S309. Details of each of these sub-steps will be described below. Note that in FIG. 21, unlike a normal information processing flowchart, where an arrow branches from one sub-step to multiple sub-steps, this means that multiple sub-steps are processed in parallel.

[0165] The autonomous operation generation unit 214 generates an autonomous operation target value (autonomous operation control information) for autonomously operating the robot arm device 10 (e.g., the arm unit 11) based on various data (external input information) output from the input information acquisition unit 212 (substep S301).

[0166] The autonomous movement correction unit 216 corrects the autonomous movement target value (autonomous movement control information) in real time based on the results of the movement simulation of the robot arm device 10 (e.g., the arm section 11) in the actual current internal body environment (real environment) and the results of the movement simulation of the robot arm device 10 (e.g., the arm section 11) in the ideal internal body environment (substep S302).

[0167] The camera predicted field of view calculation unit 218 determines a target field of view of the image capturing unit 12 for prior notification of interference and for interference avoidance based on interference prediction information acquired from the simulator unit 250 (described later) (substep S303). In detail, for example, if the image capturing unit 12 is an endoscope (not shown) with a wide-angle / cropping function, the camera predicted field of view calculation unit 218 changes the cropped field of view range 630 from the range shown on the left side of FIG. 22 to the range shown on the right side.

[0168] If the imaging unit 12 is an endoscope (not shown) with wide-angle / cropping functions, the crop field of view generation unit 230 generates a predicted image 608 that would be cropped when cropping processing is performed on the wide-angle image 606 acquired by the imaging unit 12, as shown in Figure 23, based on the cropping field of view range acquired from the camera predicted field of view calculation unit 218 (substep S304).

[0169] The visual field interlocking control calculation unit 220 generates control information (tip degree of freedom control command value, arm control command value) for operating the robot arm device 10 (e.g., the arm unit 11) while avoiding interference, based on the corrected autonomous operation target value and the target posture (interference prediction information) of the tip of the imaging unit 12 (substep S305). For example, if the imaging unit 12 is an endoscope (not shown) with a tip bending function, the visual field interlocking control calculation unit 220 generates control information that can avoid interference by moving the arm unit 11 and the tip of the imaging unit 12 in conjunction with each other, as shown in FIG. 24. In this embodiment, a predicted image 610 that will be obtained by the imaging unit 12 when interference is avoided may be generated based on the generated control information, as shown in FIG. 25.

[0170] The user instruction arbitration unit 224 corrects the control information (tip control command value, arm control command value) based on the control information (control command value) output from the field of view linkage control calculation unit 220 and the correction instructions (e.g., gaze point offset, zoom amount offset, etc.) manually input by the surgeon (user) 5067 (substep S306).

[0171] The arm control unit 226 controls the arm unit 11 based on the updated control information (arm control command value) (substep S307).

[0172] The tip control unit 228 controls the field of view, zoom amount, etc. of the imaging unit 12 based on the control information (tip control command value) (substep S308).

[0173] The autonomous action execution unit 200 proceeds to the processing of step S100 shown in FIG. 9 (sub-step S309).

[0174] ~ Presentation stage ~ Next, a control method at the presentation stage according to this embodiment will be described with reference to Figures 26 and 27. Figure 26 is a flowchart of the control method according to this embodiment, and Figure 27 is an explanatory diagram for explaining the details of the control method according to this embodiment. Note that, unlike a normal information processing flowchart, in Figure 26, where an arrow branches from one sub-step to multiple sub-steps, this means that multiple sub-steps are processed in parallel.

[0175] The presentation stage according to this embodiment is executed following the autonomous action execution stage. As shown in Fig. 26, the control method in the presentation stage can mainly include multiple sub-steps from step S401 to step S403. Each of these steps will be described in detail below.

[0176] The actual image acquisition unit 402 acquires from the control unit 210 an actual image (wide-angle image) from the imaging unit 12 and a predicted image based on the cropped field of view (step S401).

[0177] The virtual image acquisition unit 404 acquires a predicted interference image from the simulator unit 250 (step S402).

[0178] 27, the prediction result generating unit 406 simultaneously presents to the surgeon (user) 5067 one, two, or three of an actual image 610 from the imaging unit 12, a predicted image 612 based on the cut-out field of view that will be cut out, and an interference prediction image predicted by the simulator unit 250 (step S403). In this embodiment, the predicted image 612 based on the cut-out field of view that will be cut out, the interference prediction image predicted by the simulator unit 250, etc. can be presented prior to the surgeon's actions, enabling the surgeon to move their field of view, perform smooth forceps manipulation, and perform actions to avoid interference, etc.

[0179] As described above, according to the embodiment of the present disclosure, autonomous movement control information generated based on various data (external input information) can be modified based on the results of a movement simulation of the robot arm device 10 in an actual internal body environment and the results of a movement simulation of the robot arm device 10 in an ideal internal body environment, thereby making it possible to generate autonomous movement control information suitable for various internal body environments in real time. Then, by receiving assistance from the robot arm device 10 controlled in accordance with such autonomous movement control information, the surgeon can perform surgery preferably and smoothly.

[0180] <4.5 Modifications> The embodiment of the present disclosure can also be applied to a so-called master-slave operation, in which the imaging unit 12, forceps (not shown), and the like supported by the arm unit 11 can be remotely operated by an operator (user) 5067 via an input device 5047 (master console) installed in a location away from the operating room or in the operating room.

[0181] <<5. Hardware Configuration>> An information processing device such as the autonomous operation execution unit 200 according to each of the above-described embodiments is realized by a computer 1000 configured as shown in FIG. 28, for example. The autonomous operation execution unit 200 according to an embodiment of the present disclosure will be described below as an example. FIG. 28 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the autonomous operation execution unit 200 according to an embodiment of the present disclosure. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0182] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0183] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0184] The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records a program for a medical arm control method according to the present disclosure, which is an example of program data 1450.

[0185] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0186] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined computer-readable recording medium. Examples of the medium include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disk), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0187] For example, the computer 1000 may be configured as an autonomous action execution unit 200 according to an embodiment of the present disclosure. When the computer 1000 functions as the simulator unit 250, the CPU 1100 of the computer 1000 executes a program for the medical arm control method loaded onto the RAM 1200, thereby realizing the functions of the simulator unit 250, etc. The HDD 1400 may also store the program for the medical arm control method according to the present disclosure and data in the storage unit 60. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may also obtain the information processing program from another device via an external network 1550.

[0188] Furthermore, the autonomous operation execution unit 200 according to this embodiment may be applied to a system consisting of multiple devices that assumes connection to a network (or communication between devices), such as cloud computing, etc. In other words, the autonomous operation execution unit 200 according to this embodiment described above can also be realized as the medical observation system 1 according to this embodiment using multiple devices, for example.

[0189] The above shows an example of the hardware configuration of the autonomous operation execution unit 200. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. This configuration may be changed as appropriate depending on the technical level at the time of implementation.

[0190] <<6. Supplementary Information>> The above-described embodiments of the present disclosure may include, for example, an information processing method executed by the information processing device or information processing system described above, a program for causing the information processing device to function, and a non-transitory tangible medium on which the program is recorded. Furthermore, the program may be distributed via a communication line (including wireless communication) such as the Internet.

[0191] Furthermore, the steps in the information processing method of the above-described embodiment of the present disclosure do not necessarily have to be processed in the order described. For example, the steps may be processed in a different order as appropriate. Furthermore, instead of being processed chronologically, the steps may be partially processed in parallel or individually. Furthermore, the processing of each step does not necessarily have to be performed in the manner described; for example, the steps may be processed by other functional units using other methods.

[0192] Of the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information, including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0193] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0194] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0195] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0196] The present technology can also be configured as follows. (1) a control information generation unit that generates autonomous operation control information for autonomously operating the medical arm based on external input information; a simulation unit that performs a motion simulation of the medical arm; a correction unit that corrects the autonomous movement control information in real time based on a result of the movement simulation of the medical arm; A medical arm control system comprising: (2) The medical arm control system described in (1) above, wherein the correction unit corrects the autonomous operation control information in real time based on the results of a motion simulation of the medical arm in a real environment obtained from the external input information and the results of a motion simulation of the medical arm in an ideal environment. (3) a comparison unit that extracts a difference between the result of the operation simulation in the ideal environment and the result of the operation simulation in the real environment, The medical arm control system according to (2) above, wherein the correction unit corrects the autonomous operation control information in real time based on the difference. (4) The medical arm control system according to any one of (1) to (3) above, wherein the external input information includes position information and posture information of the medical arm in a real environment. (5) The medical arm control system according to any one of (1) to (4) above, wherein the medical arm supports a medical observation device. (6) The medical arm control system according to (5) above, wherein the external input information includes an image of a real environment taken by the medical observation device. (7) The medical arm control system according to (5) or (6) above, wherein the medical observation device is an endoscope. (8) The medical arm control system described in (4) above, wherein the external input information includes position information and posture information of the medical instrument in the real environment. (9) The medical arm control system described in (8) above, wherein the medical arm supports a medical instrument. (10) The medical arm control system according to any one of (1) to (9) above, wherein the external input information includes sensing data obtained by a distance measuring device. (11) The medical arm control system according to (10) above, wherein the distance measuring device measures distance using a stereo method, a ToF method, or a structured light method. (12) The medical arm control system according to any one of (1) to (11) above, wherein the autonomous operation control information includes target position information and target posture information of the medical arm. (13) The medical arm control system according to any one of (1) to (12) above, wherein the control information generation unit generates the autonomous operation control information based on a learning model obtained by machine learning. (14) a control unit that controls the medical arm based on the corrected autonomous operation control information; The medical arm control system according to any one of (1) to (13) above. (15) The medical arm control system according to any one of (1) to (14) above, further comprising the medical arm. (16) The simulation unit a real environment simulation unit that performs a motion simulation of the medical arm in the real environment by referring to a real environment map; an ideal environment simulation unit that performs a motion simulation of the medical arm in the ideal environment by referring to an ideal environment map; Including, The medical arm control system described in (2) above. (17) The medical arm control system described in (16) above, wherein the real environment simulation unit performs a motion simulation of the medical arm in the real environment using a real environment map generated based on the external input information and the autonomous motion control information generated based on the external input information. (18) The medical arm control system described in (17) above, wherein the ideal environment simulation unit performs a motion simulation of the medical arm in the ideal environment using an ideal environment map and the autonomous motion control information generated based on the corrected external input information. (19) The medical arm control system according to (18) above, further comprising an ideal environment input correction unit that corrects the external input information based on a difference between the ideal environment map and the actual environment map. (20) The medical arm control system according to (19) above, wherein the ideal environment map and the real environment map are three-dimensional map information within the human body. (twenty one) The medical arm control system described in (20) above, wherein the real environment map is generated using a SLAM (Simultaneous Localization and Mapping) method based on an image of the real environment. (twenty two) a re-simulation unit that performs a motion simulation of the medical arm in the real environment based on the corrected autonomous motion control information; The medical arm control system described in (2) above. (twenty three) a prediction unit that predicts interference with other parts based on the simulation result of the re-simulation unit, The medical arm control system according to (22) above. (twenty four) The prediction unit may include, as the interference prediction information, Position information of the other part where interference is predicted; Position information and posture information of the medical arm at the time of predicted interference; Outputs The medical arm control system according to (23) above. (twenty five) an avoidance control information generating unit that generates control information for avoiding interference based on the interference prediction information; The medical arm control system according to (24) above. (26) a predicted image generating unit that generates a predicted image based on a simulation result of the re-simulation unit; The medical arm control system according to any one of (22) to (25) above. (27) The medical arm control system described in (26) above, wherein the predicted image is presented to the user together with an image of the real environment. (28) With the medical arm control device, generating autonomous operation control information for autonomously operating the medical arm based on external input information; A motion simulation of the medical arm is performed. correcting the autonomous operation control information in real time based on the results of the operation simulation of the medical arm; A medical arm control method comprising: (29) Computer, a control information generation unit that generates autonomous operation control information for autonomously operating the medical arm based on external input information; a simulation unit that performs a motion simulation of the medical arm; a correction unit that corrects the autonomous movement control information in real time based on a result of the movement simulation of the medical arm; A program that functions as a [Explanation of symbols]

[0197] 1 Medical observation system 10 Robot arm device 11 Arm section 11a Joint 12 Imaging unit 13 Light source section 20 Control Unit 21 Image processing section 22 Imaging control unit 23 Arm control unit 25 Reception 26 Display control unit 40 Presentation device 60, 130 storage section 100 Learning Model Generation Units 110 External input information acquisition unit 120 Autonomous Action Learning Model Generation Unit 122 Autonomous operation rule generation unit 124 Ideal map generation unit 132 Learning Model 134 Rule-Based Control Model 136 Ideal Internal Environment Map 136a, 136b, 136c, 138, 600, 602, 606, 608, 610, 612 Images 140 Output section 200 Autonomous Action Execution Unit 210 Control Unit 212 Input information acquisition unit 214 Autonomous motion generation unit 216, 272 Autonomous operation correction section 218 Camera predicted field of view calculation unit 220 Visual field interlocking control calculation unit 224 User Instruction Mediation Unit 226 Arm control unit 228 Front Control Unit 230 Extraction field generation section 250 simulator unit 252 Input information receiving unit 254 Ideal Environment Map Maintenance Department 256 Internal Body Environment Map Generation Unit 258 Ideal environment input correction section 260 Ideal environment autonomous movement generation unit 262 Ideal Environment Simulation Department 264 Current environment autonomous behavior generation unit 266 Current Environmental Simulation Department 268 Autonomous operation result comparison unit 270 Autonomous motion correction parameter generation unit 274 Current Environmental Simulation Update Department 276 Interference Prediction Unit 278 Image Generation Unit 402 Actual image acquisition unit 404 Virtual Image Acquisition Unit 406 Prediction result generation unit 500 differences 502 Correction amount 504 Differences 506 Parameters 630 Field of View

Claims

1. a control information generation unit that generates autonomous operation control information for autonomously operating the medical arm based on external input information; a simulation unit that performs a motion simulation of the medical arm; a correction unit that corrects the autonomous movement control information in real time based on a result of the movement simulation of the medical arm; Equipped with the correction unit corrects the autonomous operation control information in real time based on a result of a motion simulation of the medical arm in a real environment obtained from the external input information and a result of a motion simulation of the medical arm in an ideal environment. Medical arm control system.

2. a comparison unit that extracts a difference between the result of the operation simulation in the ideal environment and the result of the operation simulation in the real environment, The medical arm control system according to claim 1 , wherein the correction unit corrects the autonomous operation control information in real time based on the difference.

3. The medical arm control system according to claim 1 or 2, wherein the external input information includes position information and posture information of the medical arm in a real environment.

4. The medical arm control system according to any one of claims 1 to 3, wherein the medical arm supports a medical observation device.

5. The medical arm control system according to claim 4 , wherein the external input information includes an image of a real environment captured by the medical observation device.

6. 6. The medical arm control system according to claim 4, wherein the medical observation device is an endoscope.

7. The medical arm control system according to claim 3 , wherein the external input information includes position information and orientation information of a medical instrument in the real environment.

8. The medical arm control system of claim 7 , wherein the medical arm supports a medical instrument.

9. The medical arm control system according to any one of claims 1 to 8, wherein the external input information includes sensing data obtained by a distance measuring device.

10. The medical arm control system according to claim 9 , wherein the distance measuring device measures distance using a stereo method, a ToF method, or a structured light method.

11. The medical arm control system according to any one of claims 1 to 10, wherein the autonomous operation control information includes target position information and target posture information of the medical arm.

12. The medical arm control system according to any one of claims 1 to 11, wherein the control information generation unit generates the autonomous operation control information based on a learning model obtained by machine learning.

13. a control unit that controls the medical arm based on the corrected autonomous operation control information; The medical arm control system according to any one of claims 1 to 12.

14. The medical arm control system according to any one of claims 1 to 13, further comprising the medical arm.

15. The simulation unit a real environment simulation unit that performs a motion simulation of the medical arm in the real environment by referring to a real environment map; an ideal environment simulation unit that performs a motion simulation of the medical arm in the ideal environment by referring to an ideal environment map; Including, The medical arm control system according to claim 1 .

16. 16. The medical arm control system according to claim 15, wherein the real environment simulation unit performs a motion simulation of the medical arm in the real environment using a real environment map generated based on the external input information and the autonomous motion control information generated based on the external input information.

17. The medical arm control system according to claim 16, wherein the ideal environment simulation unit performs a motion simulation of the medical arm in the ideal environment using an ideal environment map and the autonomous motion control information generated based on the corrected external input information.

18. The medical arm control system according to claim 17, further comprising an ideal environment input correction unit that corrects the external input information based on a difference between the ideal environment map and the actual environment map.

19. 19. The medical arm control system according to claim 18, wherein the ideal environment map and the real environment map are three-dimensional map information of the inside of a human body.

20. 20. The medical arm control system according to claim 19, wherein the real environment map is generated by a Simultaneous Localization and Mapping (SLAM) method based on an image of the real environment.

21. a re-simulation unit that performs a motion simulation of the medical arm in the real environment based on the corrected autonomous motion control information; The medical arm control system according to claim 1 .

22. a prediction unit that predicts interference with other parts based on the simulation result of the re-simulation unit, 22. The medical arm control system of claim 21.

23. The prediction unit may include, as the interference prediction information, Position information of the other part where interference is predicted; Position information and posture information of the medical arm at the time of predicted interference; Outputs 23. The medical arm control system of claim 22.

24. an avoidance control information generating unit that generates control information for avoiding interference based on the interference prediction information; 24. The medical arm control system of claim 23.

25. a predicted image generating unit that generates a predicted image based on a simulation result of the re-simulation unit; A medical arm control system according to any one of claims 21 to 24.

26. The medical arm control system according to claim 25, wherein the predicted image is presented to a user together with an image of a real environment.

27. The medical arm control device generates autonomous operation control information for autonomously operating the medical arm under control by the medical arm control device based on external input information, the medical arm control device performs a motion simulation of the medical arm, The medical arm control device corrects the autonomous operation control information in real time based on the result of the operation simulation of the medical arm. This includes: The correction includes the medical arm control device correcting the autonomous operation control information in real time based on a result of a motion simulation of the medical arm in a real environment obtained from the external input information and a result of a motion simulation of the medical arm in an ideal environment. A medical arm control method.

28. Computer, a control information generation unit that generates autonomous operation control information for autonomously operating the medical arm based on external input information; a simulation unit that performs a motion simulation of the medical arm; a correction unit that corrects the autonomous movement control information in real time based on a result of the movement simulation of the medical arm; A program that functions as the correction unit corrects the autonomous operation control information in real time based on a result of a motion simulation of the medical arm in a real environment obtained from the external input information and a result of a motion simulation of the medical arm in an ideal environment. program.

Images