Information processing device, generation method, and generation program

Abstract

An information processing apparatus (100) is provided with a generation unit which acquires an input image that is an intraoperative image, and on the basis of whether or not a substance produced during an operation is included in the input image, generates an output image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a generation method, and a generation program. [Background technology]

[0002] In the medical field, endoscopic surgery using an endoscope is widely performed. Also, various devices related to endoscopic surgery have been developed. For example, Patent Document 1 discloses a device that, when smoke is detected from a captured endoscopic image, operates an insufflation device to remove the smoke.

[0003] Patent document 2 discloses an apparatus that, when smoke is detected from a captured endoscopic image, removes the smoke from the endoscopic image through uniform signal processing, and then controls a smoke evacuation device in accordance with the smoke detection results to remove the smoke. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 11-318909 [Patent Document 2] JP 2018-157917 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, even if the presence or absence of smoke is detected, it does not detect the amount of smoke, and depending on the amount of smoke generated, it may not be possible to sufficiently remove the smoke. Also, in Patent Document 1, because the smoke is physically removed, it takes time for the smoke to be expelled and the field of vision to become clear.

[0006] In Patent Document 2, smoke is removed from endoscopic images by uniform signal processing regardless of the amount of smoke generated. The effectiveness of the signal processing is limited depending on the amount of smoke generated, and the effectiveness of smoke removal ultimately depends on the performance of the smoke evacuation device.

[0007] Therefore, the present disclosure proposes an information processing device, a generation method, and a generation program that can reduce the effects of substances generated during surgery. [Means for solving the problem]

[0008] In order to solve the above problems, one form of an information processing device according to the present disclosure includes a generation unit that acquires an input image that is an image related to surgery, and generates an output image based on whether or not the input image contains a substance that is generated during surgery. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a surgical procedure to which a surgical room system using the technical concept of the present disclosure is applied. [Diagram 2] 1 is a diagram illustrating an example of a system configuration according to a first embodiment of the present disclosure. [Diagram 3] 1 is a diagram illustrating an example of a configuration of an information processing device according to a first embodiment of the present disclosure. [Figure 4] 1A and 1B are diagrams for explaining the characteristics of smoke and mist, respectively. [Diagram 5] FIG. 13 is a diagram showing the relationship between the generation of smoke and mist and brightness and saturation. [Figure 6] 2 is a diagram illustrating an example of the configuration of a smoke removal processing unit according to the first embodiment of the present disclosure. FIG. [Figure 7] 2 is a diagram illustrating an example of the configuration of a mist removal processing unit according to the first embodiment of the present disclosure. FIG. [Figure 8] 3 is a diagram illustrating an example of the configuration of a generation unit according to the first embodiment of the present disclosure. FIG. [Figure 9] 4 is a flowchart showing a flow of basic operations of the information processing device according to the first embodiment of the present disclosure. [Figure 10] 5 is a flowchart showing a flow of an operation of a determination unit according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of a system configuration according to a second embodiment of the present disclosure. [Figure 12]FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a second embodiment of the present disclosure. [Figure 13] FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a third embodiment of the present disclosure. [Figure 14] FIG. 11 is a diagram illustrating an example of the configuration of an information processing device according to a fourth embodiment of the present disclosure. [Figure 15] FIG. 13 is a diagram illustrating an example of a system configuration according to a fifth embodiment of the present disclosure. [Figure 16] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a fifth embodiment of the present disclosure. [Figure 17] FIG. 13 is a diagram showing an example of an output image generated by the smoke removal processing according to the fifth embodiment of the present disclosure. [Figure 18] FIG. 13 is a diagram illustrating an example of the configuration of an information processing device according to a sixth embodiment of the present disclosure. [Figure 19] 13A to 13C are diagrams for explaining processing by a superimposing unit according to a sixth embodiment of the present disclosure. [Figure 20] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] The present disclosure will be described in the following order. 1. Application Examples 2. First embodiment 2.1. System Configuration According to the First Embodiment 2.2. Configuration of the information processing device according to the first embodiment 2.3. Operational flow of the information processing device 2.4. Effects of the Information Processing Device According to the First Embodiment 3. Second embodiment 3.1. System Configuration According to the Second Embodiment 3.2. Configuration of information processing device according to the second embodiment 3.3. Effects of the Information Processing Device According to the Second Embodiment 4. Third embodiment 4.1. System Configuration According to the Third Embodiment 4.2. Configuration of Information Processing Device According to Third Embodiment 4.3. Effects of the Information Processing Device According to the Third Embodiment 5. Fourth embodiment 5.1. System configuration according to the fourth embodiment 5.2. Configuration of information processing device according to the fourth embodiment 5.3. Effects of the Information Processing Device According to the Fourth Embodiment 6. Fifth embodiment 6.1. System Configuration According to Fifth Embodiment 6.2. Configuration of Information Processing Device According to Fifth Embodiment 6.3. Effects of the Information Processing Device According to the Fifth Embodiment 7. Sixth embodiment 7.1. System Configuration According to the Sixth Embodiment 7.2. Configuration of Information Processing Device According to Sixth Embodiment 7.3. Effects of the Information Processing Device According to the Sixth Embodiment 8.Hardware Configuration 9. Conclusion

[0012] <1. Application Examples> An application example of the technical idea common to each embodiment of the present disclosure will be described. FIG. 1 is a diagram showing an example of the state of surgery to which a surgery room system 5100 using the technical idea according to the present disclosure is applied. A ceiling camera 5187 and a surgery room camera 5189 are provided on the ceiling of the surgery room, and can capture the hands of an operator (doctor) 5181 who is performing treatment on an affected part of a patient 5185 on a patient bed 5183, as well as the entire state of the surgery room. The ceiling camera 5187 and the surgery room camera 5189 may be provided with a magnification adjustment function, a focal length adjustment function, a shooting direction adjustment function, and the like. A light 5191 is provided on the ceiling of the surgery room, and illuminates at least the hands of the operator 5181. The light 5191 may be capable of appropriately adjusting the amount of light emitted, the wavelength (color) of the light emitted, the light irradiation direction, and the like.

[0013] The endoscopic surgery system 5113, the patient bed 5183, the ceiling camera 5187, the surgery site camera 5189, and the lighting 5191 are connected to each other via an audiovisual controller and a surgery room control device (not shown). A centralized operation panel 5111 is provided in the surgery room, and a user can appropriately operate these devices in the surgery room via the centralized operation panel 5111.

[0014] Hereinafter, a detailed description will be given of the configuration of the endoscopic surgery system 5113. As shown in the figure, the endoscopic surgery system 5113 is composed of an endoscope 5115, other surgical tools 5131, a support arm device 5141 that supports the endoscope 5115, and a cart 5151 on which various devices for endoscopic surgery are mounted.

[0015] In endoscopic surgery, instead of cutting the abdominal wall and opening the abdomen, a cylindrical opening instrument called trocar 5139a to 5139d is punctured into the abdominal wall. Then, the lens barrel 5117 of the endoscope 5115 and other surgical instruments 5131 are inserted into the body cavity of the patient 5185 from the trocar 5139a to 5139d. In the illustrated example, as the other surgical instruments 5131, a tube 5133, an energy treatment instrument 5135, and forceps 5137 are inserted into the body cavity of the patient 5185. Here, the tube 5133 may be configured to exhaust smoke generated in the body cavity to the outside of the body cavity. On the other hand, the tube 5133 may have a function of injecting gas into the body cavity to inflate the body cavity. The energy treatment instrument 5135 is a treatment instrument that performs incision and peeling of tissue, sealing of blood vessels, etc. by high-frequency current or ultrasonic vibration. However, the surgical tool 5131 shown in the figure is merely an example, and the surgical tool 5131 may be any of various surgical tools generally used in endoscopic surgery, such as a surgical implement, a retractor, etc.

[0016] An image of the area to be operated on in the body cavity of the patient 5185 photographed by the endoscope 5115 is displayed on the display device 5155. The surgeon 5181 performs treatment such as excising the affected area using the energy treatment tool 5135 and forceps 5137 while viewing the image of the area to be operated on displayed on the display device 5155 in real time. Although not shown in the figure, the tube 5133, the energy treatment tool 5135, and the forceps 5137 are supported by the surgeon 5181 or an assistant during surgery.

[0017] (Support arm device) The support arm device 5141 includes an arm portion 5145 extending from a base portion 5143. In the example shown, the arm portion 5145 is composed of joint portions 5147a, 5147b, 5147c and links 5149a, 5149b, and is driven under the control of an arm control device 5159. The arm portion 5145 supports the endoscope 5115, and the position and attitude of the endoscope 5115 are controlled. This allows the endoscope 5115 to be stably fixed in position.

[0018] (Endoscopy) The endoscope 5115 is composed of a lens barrel 5117, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 5185, and a camera head 5119 connected to a base end of the lens barrel 5117. In the illustrated example, the endoscope 5115 is configured as a so-called rigid lens barrel having a rigid lens barrel 5117, but the endoscope 5115 may be configured as a so-called flexible lens barrel having a flexible lens barrel 5117.

[0019] An opening in which an objective lens is fitted is provided at the tip of the lens barrel 5117. A light source device 5157 is connected to the endoscope 5115, and light generated by the light source device 5157 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 5117, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 5185. The endoscope 5115 may be a direct-viewing mirror, an oblique-viewing mirror, or a side-viewing mirror.

[0020] An optical system and an imaging element are provided inside the camera head 5119, and reflected light (observation light) from an observation target is collected on the imaging element by the optical system. The imaging element photoelectrically converts the observation light to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The image signal is transmitted to a camera control unit (CCU) 5153 as RAW data. The camera head 5119 is equipped with a function for adjusting the magnification and focal length by appropriately driving the optical system.

[0021] For example, to support stereoscopic vision (3D display), a plurality of imaging elements may be provided in the camera head 5119. In this case, a plurality of relay optical systems are provided inside the lens barrel 5117 to guide observation light to each of the plurality of imaging elements.

[0022] (Various devices mounted on the cart) The CCU 5153 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the operation of the endoscope 5115 and the display device 5155 in an integrated manner. Specifically, the CCU 5153 performs various image processing, such as development processing (demosaic processing), on the image signal received from the camera head 5119 in order to display an image based on the image signal. The CCU 5153 provides the image signal subjected to the image processing to the display device 5155. The above-mentioned audiovisual controller is also connected to the CCU 5153. The CCU 5153 also provides the image signal subjected to the image processing to the audiovisual controller 5107. The CCU 5153 transmits a control signal to the camera head 5119 to control its drive. The control signal may include information on the imaging conditions, such as magnification and focal length. The information on the imaging conditions may be input via the input device 5161 or via the above-mentioned centralized operation panel 5111.

[0023] The display device 5155 displays an image based on an image signal that has been subjected to image processing by the CCU 5153 under the control of the CCU 5153. When the endoscope 5115 is compatible with high-resolution photography such as 4K (3840 horizontal pixels x 2160 vertical pixels) or 8K (7680 horizontal pixels x 4320 vertical pixels) and / or compatible with 3D display, the display device 5155 may be capable of displaying high resolution and / or 3D display correspondingly. When compatible with high-resolution photography such as 4K or 8K, a display device 5155 with a size of 55 inches or more can be used to provide a more immersive feeling. In addition, multiple display devices 5155 with different resolutions and sizes may be provided depending on the application.

[0024] The light source device 5157 is composed of a light source such as an LED (light emitting diode) and supplies the endoscope 5115 with irradiation light when photographing the surgical site.

[0025] The arm control device 5159 is constituted by a processor such as a CPU, and operates according to a prescribed program to control the driving of the arm portion 5145 of the support arm device 5141 according to a prescribed control method.

[0026] The input device 5161 is an input interface for the endoscopic surgery system 5113. A user can input various pieces of information and instructions to the endoscopic surgery system 5113 via the input device 5161. For example, the user inputs various pieces of information related to surgery, such as patient physical information and information about the surgical procedure, via the input device 5161. In addition, for example, the user inputs, via the input device 5161, an instruction to drive the arm unit 5145, an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 5115, an instruction to drive the energy treatment tool 5135, etc.

[0027] The type of the input device 5161 is not limited, and the input device 5161 may be any of various known input devices. For example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5171, and / or a lever may be applied as the input device 5161. When a touch panel is used as the input device 5161, the touch panel may be provided on a display surface of the display device 5155.

[0028] Alternatively, the input device 5161 is a device worn by a user, such as a glasses-type wearable device or a head mounted display (HMD), and various inputs are made according to the user's gestures and gaze detected by these devices. The input device 5161 also includes a camera capable of detecting the user's movements, and various inputs are made according to the user's gestures and gaze detected from the image captured by the camera. The input device 5161 also includes a microphone capable of collecting the user's voice, and various inputs are made by voice via the microphone. In this way, the input device 5161 is configured to be able to input various information in a non-contact manner, and thus it becomes possible for a user (e.g., the surgeon 5181) belonging to a clean area to operate a device belonging to an unclean area in a non-contact manner. In addition, the user can operate the device without taking his / her hands off the surgical tool he / she is holding, which improves the user's convenience.

[0029] The treatment tool control device 5163 controls the driving of the energy treatment tool 5135 for cauterizing tissue, incising, sealing blood vessels, etc. The smoke exhaust device 5165 sends gas into the body cavity of the patient 5185 through the tube 5133 to inflate the body cavity in order to ensure the field of view of the endoscope 5115 and the working space of the surgeon. The smoke exhaust device 5165 also has a function of exhausting smoke generated in the body cavity in order to ensure the field of view of the endoscope 5115. The recorder 5167 is a device capable of recording various information related to the surgery. The printer 5169 is a device capable of printing various information related to the surgery in various formats such as text, image, and graph.

[0030] Below, the particularly characteristic configuration of the endoscopic surgery system 5113 will be described in further detail.

[0031] (Support arm device) The support arm device 5141 includes a base 5143 serving as a base, and an arm 5145 extending from the base 5143. In the illustrated example, the arm 5145 includes a plurality of joints 5147a, 5147b, and 5147c, and a plurality of links 5149a and 5149b connected by the joint 5147b. However, in FIG. 1, for the sake of simplicity, the configuration of the arm 5145 is simplified. In practice, the shapes, number, and arrangement of the joints 5147a to 5147c and the links 5149a and 5149b, as well as the directions of the rotation axes of the joints 5147a to 5147c, can be appropriately set so that the arm 5145 has a desired degree of freedom. For example, the arm 5145 can be preferably configured to have six or more degrees of freedom. This allows the endoscope 5115 to be moved freely within the movable range of the arm portion 5145, making it possible to insert the telescope barrel 5117 of the endoscope 5115 into the body cavity of the patient 5185 from a desired direction.

[0032] The joints 5147a to 5147c are provided with actuators, and the joints 5147a to 5147c are configured to be rotatable around a predetermined rotation axis by driving the actuators. The drive of the actuators is controlled by the arm control device 5159, whereby the rotation angles of the joints 5147a to 5147c are controlled, and the drive of the arm 5145 is controlled. This allows control of the position and attitude of the endoscope 5115. At this time, the arm control device 5159 can control the drive of the arm 5145 by various known control methods such as force control or position control.

[0033] For example, the surgeon 5181 may appropriately input an operation via the input device 5161 (including the foot switch 5171), and the drive of the arm unit 5145 may be appropriately controlled by the arm control device 5159 in response to the operation input, thereby controlling the position and posture of the endoscope 5115. Through this control, the endoscope 5115 at the tip of the arm unit 5145 may be moved from an arbitrary position to an arbitrary position, and then fixedly supported at the position after the movement. The arm unit 5145 may be operated in a so-called master-slave manner. In this case, the arm unit 5145 may be remotely operated by a user via the input device 5161 installed at a location away from the operating room.

[0034] Furthermore, when force control is applied, the arm control device 5159 may perform so-called power assist control, in which the actuators of the joints 5147a to 5147c are driven so that the arm unit 5145 moves smoothly in response to an external force from the user. This allows the arm unit 5145 to be moved with a relatively light force when the user moves the arm unit 5145 while directly touching the arm unit 5145. This makes it possible to move the endoscope 5115 more intuitively and with a simpler operation, improving the convenience for the user.

[0035] Generally, in endoscopic surgery, the endoscope 5115 is supported by a doctor called a scopist. By contrast, by using the support arm device 5141, the position of the endoscope 5115 can be fixed more reliably without relying on human hands, so that an image of the surgical site can be obtained stably and the surgery can be performed smoothly.

[0036] The arm control device 5159 does not necessarily have to be provided on the cart 5151. Also, the arm control device 5159 does not necessarily have to be one device. For example, the arm control device 5159 may be provided on each of the joints 5147a to 5147c of the arm section 5145 of the support arm device 5141, and the drive control of the arm section 5145 may be realized by the multiple arm control devices 5159 cooperating with each other.

[0037] (Light source device) The light source device 5157 supplies the endoscope 5115 with irradiation light for photographing the surgical site. The light source device 5157 is composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. In this case, when the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the light source device 5157 can adjust the white balance of the captured image. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation target with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 5119 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0038] Furthermore, the light source device 5157 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 5119 in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner and synthesizing the images, it is possible to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0039] The light source device 5157 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, an excitation light is irradiated to a body tissue and the fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and an excitation light corresponding to the fluorescence wavelength of the reagent is irradiated to the body tissue to obtain a fluorescent image. The light source device 5157 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0040] 2. First embodiment <<2.1. System configuration according to the first embodiment>> Next, a first embodiment of the present disclosure will be described in detail. Fig. 2 is a diagram showing an example of a system configuration according to the first embodiment of the present disclosure. As shown in Fig. 2, this system includes an imaging device 10, a display device 5155, and an information processing device 100. The imaging device 10, the display device 5155, and the information processing device 100 are connected to each other via a network 20.

[0041] The imaging device 10 is a device that captures an in vivo image of an observation target inside a living body. The imaging device 10 may be, for example, an endoscope 5115 as described in FIG. 1. The imaging device 10 includes an imaging unit 11 and a communication unit 12.

[0042] The imaging unit 11 has a function of capturing an in vivo image of an observation subject in the living body. The imaging unit 11 according to this embodiment includes an imaging element such as a CCD (Charge Coupled device) or a CMOS (Complementary MOS). The imaging unit 11 captures an in vivo image at a predetermined frame rate (FPS: Frames Per Second).

[0043] Here, the in-vivo image according to the present embodiment broadly includes images acquired from a biological perspective for clinical, medical, and experimental purposes (biological imaging), and the imaging subject is not limited to a human being.

[0044] The communication unit 12 has a function of performing information communication with the information processing device 100 via the network 20. For example, the communication unit 12 transmits each in-vivo image captured by the imaging unit 11 to the information processing device 100 in chronological order.

[0045] The information processing device 100 receives each in vivo image in chronological order from the imaging device 10. In the following description, the in vivo image received from the imaging device 10 is referred to as an "input image." The information processing device 100 determines whether or not the input image contains a substance that occurs during surgery, and generates an output image based on the determination result and the input image. For example, substances that occur during surgery include smoke and mist. The input image is also called a "medical image" or an "intraoperative image."

[0046] The information processing device 100 transmits the output image to the display device 5155. As described later, when the input image includes smoke or mist, the information processing device 100 generates an output image from which the smoke or mist is excluded. The information processing device 100 may be, for example, the CCU 5153 as described in FIG. 1.

[0047] The display device 5155 receives an output image from the information processing device 100 and displays the received output image. The rest of the description of the display device 5155 is the same as the description of the display device 5155 in FIG.

[0048] <<2.2. Configuration of the information processing device according to the first embodiment>> Next, the information processing device 100 according to the first embodiment of the present disclosure will be described in detail. Fig. 3 is a diagram showing a configuration example of the information processing device according to the first embodiment of the present disclosure. As shown in Fig. 3, the information processing device 100 has a storage unit 101, a determination unit 102, a smoke removal processing unit 103, a mist removal processing unit 104, and a generation unit 105.

[0049] Every time the information processing device 100 receives an input image from the imaging device 10, it inputs the input image to the determination unit 102, the smoke removal processing unit 103, and the mist removal processing unit 104. Although not shown in FIG. 3, the information processing device 100 is assumed to have a communication unit that communicates information with the imaging device 10 and the display device 5155 via the network 20.

[0050] ((Storage unit 101)) The storage unit 101 is a storage device that stores information on the latest output image generated by the generation unit 105. The output image is an image in which no smoke or mist is generated (or from which the smoke and mist have been removed). The output image stored in the storage unit 101 is updated every time a new output image is output from the generation unit 105.

[0051] The storage unit 101 corresponds to a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), a flash memory, etc., or a storage device such as a hard disk drive (HDD), etc. The storage unit 101 may be either a volatile memory or a non-volatile memory, or may use both.

[0052] ((Judgment unit 102)) The determination unit 102 is a processing unit that determines whether or not smoke or mist is included in an input image based on the input image. If the determination unit 102 determines that smoke or mist is included, the determination unit 102 determines the amount of smoke and mist generated. In addition, the determination unit 102 calculates the probability of smoke and mist generation, respectively. The probability of smoke and mist generation corresponds to the ratio of smoke to mist.

[0053] Here, the characteristics of smoke and mist will be described. Fig. 4 is a diagram for explaining the characteristics of smoke and mist. In Fig. 4, image 25 is an input image in which smoke and mist are not generated. Image 25a is an input image in which smoke is generated. Image 25b is an input image in which mist is generated.

[0054] When smoke occurs, it spreads relatively uniformly, just like fog, as shown in image 25a. The smoked area has the characteristic that it becomes whitish overall and the contrast of the background is reduced because light is reflected and the transmittance is reduced in the smoked area.

[0055] When mist occurs, as shown in Image 25b, it becomes whitish like smoke and the contrast of the background decreases. Since mist is basically a collection of water vapor and water droplets, the difference in transmittance between the areas with and without water droplets becomes large, and areas of the background that are not visible are created like a mottled pattern.

[0056] Next, the relationship between the generation of smoke and mist and the brightness and saturation will be described. FIG. 5 is a diagram showing the relationship between the generation of smoke and mist and the brightness and saturation. In the graph shown in FIG. 5, the horizontal axis corresponds to time. The vertical axis corresponds to the level (value) of brightness or saturation. Line 26a shows the relationship between brightness and time. Line 26b shows the relationship between saturation and time.

[0057] As shown in FIG. 5, when smoke (or mist) is generated at time t and the amount of smoke (or mist) generated increases over time, the luminance level increases and the saturation level decreases as the amount of smoke (or mist) generated increases.

[0058] An example of a process in which the determination unit 102 determines whether or not smoke or mist is included in an input image will be described. The determination unit 102 calculates a reference value of luminance and a reference value of saturation based on the output image stored in the storage unit 101. The determination unit 102 converts each pixel value of the output image into luminance and saturation. The determination unit 102 calculates the average value of each luminance of the output image as the reference value of luminance. The determination unit 102 calculates the average value of each saturation of the output image as the reference value of saturation.

[0059] When the determination unit 102 receives an input of an input image, it converts each pixel value included in the input image into luminance and saturation. For example, when the average value of each luminance of the input image is less than a reference value of luminance and the average value of each saturation of the input image is less than a reference value of saturation, the determination unit 102 determines that the input image contains smoke or mist.

[0060] When the determination unit 102 determines that smoke or mist is contained in the input image, it executes a process of determining the amount of smoke and mist generated, and a process of determining the probability of smoke and mist generation.

[0061] The following describes an example of a process for determining the amount of smoke and mist generated by the determination unit 102. For example, the determination unit 102 divides an input image into a plurality of blocks, and calculates changes in luminance and saturation over time for each block.

[0062] For example, the determination unit 102 determines whether a block BO ij and the luminance of the block BI of the input image. ij The difference between the brightness of the BO and the brightness of the ij BI indicates the block in the i-th row and j-th column of the divided blocks of the output image. ij indicates the block in the i-th row and j-th column of the divided blocks of the input image.

[0063] The determination unit 102 determines the block BO of the output image. ij and the saturation of the block BI of the input imageij The difference between the saturation of the first color and the saturation of the second color is calculated as the change in saturation over time.

[0064] The determination unit 102 compares each block of the output image with each block of the input image, and calculates the change in luminance and saturation over time for each block. The determination unit 102 determines, among the blocks of the input image, blocks in which the change in luminance and saturation over time is equal to or greater than a threshold value, as blocks in which smoke or mist has occurred.

[0065] For example, the determination unit 102 determines the amount of smoke and mist generated based on a "generation amount determination table (not shown)" that defines the relationship between the ratio of blocks in which smoke or mist is generated among all blocks of the input image, the time change in brightness and saturation, and the amount of smoke and mist generated. The information in the generation amount determination table is set in advance.

[0066] Next, an example of a process in which the determination unit 102 determines the probability (ratio) of smoke and mist generation will be described. The determination unit 102 calculates a dynamic range for each block of the input image. For example, the determination unit 102 scans the luminance (or pixel value) included in one block, identifies the maximum luminance value and the minimum luminance value, and calculates the difference between the maximum luminance value and the minimum luminance value as the dynamic range.

[0067] The determination unit 102 determines the probability of smoke and mist occurring for one block based on an "occurrence ratio specification table (not shown)" that defines the relationship between the dynamic range and the probability of smoke and mist occurring. For example, since mist tends to have a larger local contrast in an image than smoke, the larger the dynamic range, the higher the probability of mist occurring compared to the probability of smoke occurring.

[0068] The determination unit 102 executes a process of calculating the probability of smoke and mist occurrence for each block where smoke or mist occurs. The determination unit 102 specifies a representative value of the probability of smoke and mist occurrence based on the calculated probability of smoke and mist occurrence. For example, the determination unit 102 specifies an average value or a median value of the probability of smoke and mist occurrence as the representative value of the probability of smoke and mist occurrence.

[0069] For example, the determination unit 102 adjusts the probability P1 of smoke generation and the probability P2 of mist generation so that the sum of the probability P1 of smoke generation and the probability P2 of mist generation becomes 100%.

[0070] The determination unit 102 outputs the determination result to the generation unit 105. The determination result includes information on whether or not smoke or mist is occurring in the input image. If smoke or mist is occurring, the determination result further includes the amount of smoke and mist occurring and a representative value of the probability of smoke and mist occurring. In the following description, the representative value of the probability of smoke and mist occurring is simply referred to as the probability of smoke and mist occurring.

[0071] ((Smoke removal processing unit 103)) The smoke removal processing unit 103 is a processing unit that generates a smoke-removed image in which smoke has been reduced or removed from an input image. In the following description, "reducing or removing smoke from an input image" will be expressed as "removing smoke from an input image" as appropriate. FIG. 6 is a diagram showing an example of the configuration of a smoke removal processing unit according to the first embodiment of the present disclosure. As shown in FIG. 6, the smoke removal processing unit 103 has a deterioration estimation unit 31 and a deterioration correction unit 32.

[0072] The degradation estimation unit 31 is a processing unit that estimates the degradation of an input image based on an input image and an output image. The degradation estimation unit 31 outputs the degradation estimation result to the degradation correction unit 32. For example, the degradation estimation unit 31 executes a histogram conversion process and a correction amount map calculation process. The histogram conversion process and the correction amount map calculation process executed by the degradation estimation unit 31 will be described below.

[0073] An example of the histogram conversion process executed by the degradation estimation unit 31 will be described. The degradation estimation unit 31 converts the histogram h S (s j ) is the histogram of the output image (target image) h T (t j ), where s j indicates the j-th pixel value in the input image. j indicates the j-th pixel value in the output image. For example, pixel values ​​in the input image and the output image range from 0 to 255.

[0074] The degradation estimation unit 31 normalizes the histogram by the number of pixels to obtain a probability density function. For example, the probability density function p S (s j ) is defined by equation (1). The degradation estimation unit 31 calculates the probability density function p S (s j ) is calculated.

[0075]

number

[0076] Probability density function of the output image p T (t j ) is defined by equation (2). The deterioration estimation unit 31 calculates the probability density function p T (t j ) is calculated.

[0077]

number

[0078] After calculating the probability density function, the degradation estimation unit 31 calculates the cumulative distribution function of the probability density function. For example, the cumulative distribution function F S (s k ) is defined by equation (3). The deterioration estimation unit 31 calculates the cumulative distribution function F S (sk If the pixel values ​​of the input image range from 0 to 255, k=255.

[0079]

number

[0080] Cumulative distribution function of the output image F T (t k ) is defined by equation (4). The deterioration estimation unit 31 calculates the cumulative distribution function F T (t k If the pixel values ​​of the output image range from 0 to 255, k=255.

[0081]

number

[0082] The deterioration estimation unit 31 is F T (s k ) inverse function F T -1 (s k ) and F S (s k )=F T (s k For example, the degradation estimation unit 31 converts each pixel value s of the input image into j (j=0 to 255) is calculated based on the equation (5) as the output pixel value o j The histogram-converted image H(I) is generated by converting the histograms into

[0083]

number

[0084] Here, the deterioration estimation unit 31 does not necessarily need to perform the above-mentioned histogram conversion process on the entire image, but may perform the process on a specific region of the image or on a grid basis.

[0085] An example of the correction amount map calculation process executed by the deterioration estimation unit 31 will be described. The deterioration estimation unit 31 calculates the correction amount map M based on equation (6). As shown in equation (6), the deterioration estimation unit 31 calculates the difference between the histogram converted image (I) and the input image I to calculate the contrast correction amount map M.

[0086] M = H(I) - I (6)

[0087] Furthermore, the degradation estimation unit 31 generates a shaped correction amount map F(I,M) by shaping the contrast correction amount map with a guided filter using the input image as a guide image. The shaped correction amount map F(I,M) is the correction amount map M shaped to match the edges of the input image I. In this way, by using the shaped correction amount map F(I,M), it is possible to prevent image degradation around edges that may occur when the positions of the correction amount map M and the input image I are misaligned.

[0088] The deterioration estimation unit 31 may generate the shaped correction amount map F(I,M) by using a guided filter described in any one of Non-Patent Documents 1, 2, and 3 shown below.

[0089] Non-patent document 1: Kopf, Johannes, et al. "Joint bilateral upsampling." ACM Transactions on Graphics (ToG). Vol. 26. No. 3. ACM, 2007. Non-patent document 2: He, Kaiming, Jian Sun, and Xiaoou Tang. "Guided image filtering." European conference on computer vision. Springer, Berlin, Heidelberg, 2010. Non-Patent Document 3: Gastal, Eduardo SL, and Manuel M. Oliveira. "Domain transform for edge-aware image and video processing." ACM Transactions on Graphics (ToG). Vol. 30. No. 4. ACM, 2011.

[0090] The deterioration estimation unit 31 outputs the formed correction amount map F(I,M) as an estimation result of the deterioration to the deterioration corrector 32. In the formed correction amount map F(I,M), a pixel value of each pixel is defined.

[0091] The degradation correction unit 32 is a processing unit that corrects the input image based on the degradation estimation result to generate a smoke-removed image. For example, the degradation correction unit 32 generates a smoke-removed image O based on the equation (7). A Equation (7) means that a process of adding the pixel value of a pixel at the same position of the shaped correction amount map F(I,M) to the pixel value of the pixel of the input image I is executed for each pixel. The degradation correction unit 32 outputs information on the smoke-removed image to the generation unit 105.

[0092] O A =F(I,M)+I (7)

[0093] ((Mist removal processing unit 104)) The mist removal processing unit 104 is a processing unit that generates a mist-removed image in which mist is reduced or removed from an input image. In the following description, "reducing or removing mist from an input image" is appropriately expressed as "removing mist from an input image." FIG. 7 is a diagram showing an example of the configuration of a mist removal processing unit according to the first embodiment of the present disclosure. As shown in FIG. 7, the mist removal processing unit 104 has an occurrence region identification unit 41, a first deterioration correction unit 42, a deterioration estimation unit 43, and a second deterioration correction unit 44.

[0094] The occurrence region identifying unit 41 is a processing unit that compares an input image with an output image and determines the region of the input image where mist is occurring. For example, in the same manner as the determining unit 102, the occurrence region identifying unit 41 determines, among the blocks of the input image, a block where the change in luminance and saturation over time is equal to or greater than a threshold, as a block where smoke or mist is occurring. In addition, the occurrence region identifying unit 41 determines, among the blocks where smoke or mist is occurring, a block where the luminance is equal to or greater than a threshold Th Y Blocks in which this is the case or more are identified as areas in which mist has occurred.

[0095] The mist generation region identifying unit 41 may identify the region where the mist is generated by other processing. For example, the mist generation region identifying unit 41 may divide the input image into a plurality of blocks, and identify the region where the mist is generated by dividing the input image into a plurality of blocks. Y A block in which this is the case or more may be identified as an area in which mist has occurred.

[0096] The generation region identifying section 41 outputs information on the region where mist is generated and information on the region where mist is not generated to the first deterioration correction section 42. In addition, the generation region identifying section 41 outputs information on the input image to the first deterioration correction section 42.

[0097] The first deterioration correction unit 42 is a processing unit that corrects areas in the input image where mist occurs using information on areas where mist does not occur. The first deterioration correction unit 42 divides the input image into a plurality of blocks, and classifies the divided plurality of blocks into blocks where mist occurs and blocks where mist does not occur. In the following description, the blocks where mist occurs are referred to as first blocks, and the blocks where mist does not occur are referred to as second blocks.

[0098] The first degradation correction unit 42 selects a first block, and selects a second block located within a predetermined distance from the selected first block. The first degradation correction unit 42 adjusts the contrast of the selected first block so that the contrast is the same as the contrast of the selected second block. When there are multiple second blocks located within a predetermined distance from the selected first block, the first degradation correction unit 42 may calculate an average value of the contrasts of the multiple second blocks, and adjust the contrast of the first block so that it is the same as the average value of the contrasts of the multiple second blocks.

[0099] The first degradation corrector 42 corrects the input image by repeatedly executing the above process for each first block included in the input image.

[0100] The first deterioration correcting unit 42 outputs the corrected input image to the second deterioration correcting unit 44. In the following description, the corrected input image will be referred to as a "corrected image."

[0101] The deterioration estimation unit 43 is a processing unit that estimates the deterioration of the input image based on the input image and the output image. The deterioration estimation unit 43 outputs the deterioration estimation result to the second deterioration correction unit 44. The process of the deterioration estimation unit 43 is similar to the process of the deterioration estimation unit 31. For example, the deterioration estimation result that the deterioration estimation unit 43 outputs to the second deterioration correction unit 44 becomes a formed correction amount map F(I,M).

[0102] The second deterioration correction unit 44 is a processing unit that corrects the corrected image based on the deterioration estimation result to generate a mist-removed image. For example, the second deterioration correction unit 44 generates a mist-removed image O based on the formula (8). B In equation (8), the corrected image I B This means that the process of adding the pixel value of the pixel at the same position in the shaped correction amount map F(I,M) to the pixel value of the pixel in the image is executed for each pixel. The second deterioration correction unit 44 outputs information on the mist-removed image to the generation unit 105.

[0103] O B =F(I,M)+IB (8)

[0104] ((Generation unit 105)) The generation unit 105 is a processing unit that generates an output image based on the determination result by the determination unit 102 and the input image. The generation unit 105 outputs information on the output image to the display device 5155. The influence of smoke or mist is removed from the entire output image. FIG. 8 is a diagram showing an example of the configuration of the generation unit according to the first embodiment of the present disclosure. As shown in FIG. 8, the generation unit 105 has a first blending ratio calculation unit 51, a first blending processing unit 52, a second blending ratio calculation unit 53, and a second blending processing unit 54.

[0105] The first blending ratio calculation unit 51 is a processing unit that calculates the blending ratio α between the smoke-removed image and the mist-removed image based on the determination result of the determination unit 102. The first blending ratio calculation unit 51 sets the probability of smoke and mist occurrence as the blending ratio α. For example, if the probability P1 of smoke occurrence and the probability P2 of mist occurrence are adjusted to sum to 100%, the blending ratio α between smoke and mist is P1:P2. The first blending ratio calculation unit 51 outputs information on the blending ratio α to the first blending processing unit 52.

[0106] The first blending processor 52 is a processor that generates a processed image by blending (combining) the smoke-removed image and the mist-removed image based on the blending ratio α. The first blending processor 52 outputs information on the processed image to the second blending processor 54.

[0107] For example, the pixel value of the pixel in the i-th row and j-th column of the processed image is S3 ij The pixel value of the pixel in the i-th row and j-th column of the smoke-removed image is S1 ij The pixel value of the pixel in the i-th row and j-th column of the mist-removed image is S2 ij In this case, the first blending unit 52 uses the pixel value S3 of the processing pixel as ij is calculated using equation (9).

[0108] S1 ij =P1 / (P1+P2)×S1ij +P2 / (P1+P2)×S2 ij (9)

[0109] The first blending processor 52 generates a processed image by calculating the pixel values ​​of each pixel of the processed image based on equation (9).

[0110] The second blend ratio calculation unit 53 is a processing unit that calculates a blend ratio β between the processed image and the input image based on the determination result of the determination unit 102. For example, the second blend ratio calculation unit 53 calculates the blend ratio β based on a "blend ratio specification table (not shown)" that defines the relationship between the amount of smoke and mist generated and the blend ratio β.

[0111] For example, the blend ratio β between the processed image and the input image will be described as P3:P4. In the blend ratio specification table, the ratio P3 of the processed image is set to be greater than the ratio of the input image P4 as the amount of smoke and mist generated increases. The second blend ratio calculation unit 53 outputs information on the blend ratio β to the second blend processing unit 54.

[0112] The second blending unit 54 is a processing unit that generates an output image by blending (combining) the processed image and the input image based on the blending ratio β. For example, the pixel value of the pixel in the i-th row and j-th column of the processed image is S3 ij The pixel value of the pixel in the i-th row and j-th column of the input image is S4 ij The pixel value of the pixel in the i-th row and j-th column of the output image is S5 ij In this case, the second blending unit 54 uses the pixel value S5 of the output image. ij is calculated using equation (10).

[0113] S5 ij =P3 / (P3+P4)×S3 ij +P4 / (P3+P4)×S4 ij (10)

[0114] The second blending processor 54 generates an output image by calculating the pixel values ​​of each pixel of the processed image based on equation (10).

[0115] <<2.3. Operational Flow of Information Processing Device 100>> Next, a description will be given of the flow of operations of the information processing device 100 according to the first embodiment of the present disclosure. Fig. 9 is a flowchart showing the flow of basic operations of the information processing device 100 according to the first embodiment of the present disclosure.

[0116] 9, the information processing device 100 receives an input image from the imaging device 10 (step S101). The determination unit 102 of the information processing device 100 executes a determination process (step S102).

[0117] The smoke removal processing unit 103 of the information processing device 100 performs a smoke removal process on the input image to generate a smoke-removed image (step S103). The mist removal processing unit 104 of the information processing device 100 performs a mist removal process on the input image to generate a mist-removed image (step S104).

[0118] The generation unit 105 of the information processing device 100 blends the smoke-removed image and the mist-removed image using a blend ratio α to generate a processed image (step S105). The generation unit 105 blends the processed image and the input image using a blend ratio β to generate an output image (step S106).

[0119] The generating unit 105 registers the output image in the storage unit 101 and outputs the output image to the display device 5155 (step S107). If the information processing device 100 continues the process (step S108, Yes), the information processing device 100 proceeds to step S101. On the other hand, if the information processing device 100 does not continue the process (step S108, No), the information processing device 100 ends the process.

[0120] Next, a description will be given of the flow of operation of the determination process shown in step S102 in Fig. 9. Fig. 10 is a flowchart showing the flow of operation of the determination unit according to the first embodiment of the present disclosure.

[0121] The determination unit 102 of the information processing device 100 converts pixel values ​​of an input image into luminance and saturation (step S201). The determination unit 102 divides the input image into a plurality of blocks (step S202).

[0122] The determination unit 102 calculates the change in luminance and chroma over time for each block (step S203). The determination unit 102 estimates the amount of smoke and mist generated from the change in luminance and chroma over time and the area (step S204).

[0123] The determination unit 102 calculates the dynamic range for each block (step S205). The determination unit 102 estimates the occurrence probability of each of smoke and mist (step S206). The determination unit 102 outputs the determination result to the generation unit 105 (step S207).

[0124] <<2.4. Effects of the information processing device according to the first embodiment>> According to the information processing device 100 of the first embodiment of the present disclosure, it is determined whether or not the input image contains smoke or mist based on the input image, and an output image from which the smoke or mist has been removed is generated based on the determination result and the input image. As a result, even if smoke or mist is generated during endoscopic surgery and visibility becomes poor, it is possible to ensure a clearer view by using the information processing device 100 regardless of the amount of smoke or mist generated.

[0125] According to the information processing device 100, the smoke removal processing unit 103 adjusts the contrast of the input image so that it is the same as the contrast of an output image in which no smoke or mist is generated, thereby generating a smoke-removed image. This makes it possible to appropriately remove smoke contained in the input image.

[0126] According to the information processing device 100, the mist removal processing unit 104 adjusts the contrast of the input image so that the contrast of the mist generation area is the same as the contrast of the area where the mist is not generated, and then further adjusts the contrast of the input image so that the contrast is the same as the contrast of the output image where neither smoke nor mist is generated. By adjusting the contrast of the input image in two stages in this way, even mottled mist, which has different characteristics from smoke, can be appropriately removed from the input image.

[0127] According to the information processing device 100, the probability of smoke or mist occurrence is determined based on the input image. This makes it possible to synthesize the smoke-removed image and the mist-removed image using a blend ratio α based on the probability of smoke or mist occurrence, and to appropriately remove the smoke and mist contained in the input image.

[0128] According to the information processing device 100, the amount of smoke or mist generated is determined based on the input image. This makes it possible to combine the processed image and the input image using a blend ratio β based on the amount of smoke or mist generated, and to appropriately remove the smoke and mist contained in the input image.

[0129] Furthermore, according to the information processing device 100, since the processing related to removing smoke and mist is executed on the input image, when smoke or mist occurs, the effect of reducing the smoke or mist can be obtained instantaneously.

[0130] 3. Second embodiment <<3.1. System configuration according to the second embodiment>> Next, a second embodiment of the present disclosure will be described in detail. Fig. 11 is a diagram showing an example of a system configuration according to the second embodiment of the present disclosure. As shown in Fig. 11, this system includes an imaging device 10, a device-in-use monitoring device 60, a display device 5155, and an information processing device 200. The imaging device 10, the device-in-use monitoring device 60, the display device 5155, and the information processing device 200 are connected to each other via a network 20.

[0131] The description of the imaging device 10 and the display device 5155 is similar to the description of the imaging device 10 and the display device 5155 described in FIG.

[0132] The device-in-use monitoring device 60 is connected to an electric scalpel, an ultrasonic coagulating and cutting device (not shown), and monitors whether the electric scalpel or the ultrasonic coagulating and cutting device is being used. The device-in-use monitoring device 60 may be, for example, the treatment tool control device 5163 described in FIG. 1. The device-in-use monitoring device 60 has a monitoring unit 61 and a communication unit 62.

[0133] The monitoring unit 61 is a processing unit that monitors the usage status of the electric scalpel and the ultrasonic coagulating and cutting device. For example, when the monitoring unit 61 receives a control signal for starting usage from a start button of the electric scalpel or the ultrasonic coagulating and cutting device, the monitoring unit 61 determines that the electric scalpel or the ultrasonic coagulating and cutting device is in use. The monitoring unit 61 generates usage device information. The usage device information includes information on whether the electric scalpel is in use or not, and information on whether the ultrasonic coagulating and cutting device is in use or not.

[0134] For example, an electric scalpel uses heat generated by a high-frequency current to stop bleeding or make an incision on an affected area of ​​a patient 5185. An electric scalpel is characterized in that it easily generates smoke because it burns the treated area.

[0135] The ultrasonic coagulating and cutting device uses friction caused by ultrasonic vibration to coagulate and cut the affected area of ​​the patient 5185. The ultrasonic coagulating and cutting device has a feature that mist is easily generated by ultrasonic vibration.

[0136] The communication unit 62 has a function of communicating information with the information processing device 200 via the network 20. For example, the communication unit 62 transmits the device usage information generated by the monitoring unit 61 to the information processing device 200.

[0137] <<3.2. Configuration of the information processing device according to the second embodiment>> Next, the information processing device 200 according to the second embodiment of the present disclosure will be described in detail. Fig. 12 is a diagram showing a configuration example of the information processing device according to the second embodiment of the present disclosure. As shown in Fig. 12, the information processing device 200 has a storage unit 201, a determination unit 202, a smoke removal processing unit 203, a mist removal processing unit 204, and a generation unit 205.

[0138] Every time the information processing device 200 receives an input image from the imaging device 10, it inputs the input image to the determination unit 202, the smoke removal processing unit 203, and the mist removal processing unit 204. Every time the information processing device 200 receives device use information from the device use monitoring device 60, it inputs device use information to the determination unit 202. Although not shown in FIG. 12, the information processing device 200 is assumed to have a communication unit that communicates information with the imaging device 10, the device use monitoring device 60, and the display device 5155 via the network 20.

[0139] ((Storage unit 201))) The storage unit 201 is a storage device that stores information on the latest output image generated by the generation unit 205. The output image is an image in which no smoke or mist is generated (or from which the smoke and mist have been removed). The output image stored in the storage unit 201 is updated every time a new output image is output from the generation unit 205.

[0140] The storage unit 201 corresponds to a semiconductor memory element such as a RAM, a ROM, or a flash memory, or a storage device such as a HDD, etc. The storage unit 201 may be either a volatile memory or a non-volatile memory, or may use both.

[0141] ((Judgment unit 202)) The determination unit 202 is a processing unit that determines whether or not smoke or mist is included in the input image based on the device information and the input image. If the determination unit 202 determines that smoke or mist is included, the determination unit 202 determines the amount of smoke and mist generated. In addition, the determination unit 202 calculates the probability of smoke and mist generation, respectively. The probability of smoke and mist generation corresponds to the ratio of smoke to mist.

[0142] An example of a process in which the determination unit 202 determines whether or not smoke or mist is included in an input image will be described below. For example, the determination unit 202 determines that smoke or mist is included in the input image when the device information includes information indicating that an electric scalpel or an ultrasonic coagulating and cutting device is being used.

[0143] The determination unit 202 may determine that the input image contains smoke or mist based on the input image and may determine that the input image contains smoke or mist when the device information includes information indicating that an electric scalpel or an ultrasonic coagulating and cutting device is being used. The process by which the determination unit 202 determines that the input image contains smoke or mist based on the input image is similar to the process by the determination unit 102 in the first embodiment.

[0144] When the determining unit 202 determines that smoke or mist is contained in the input image, it executes a process of determining the amount of smoke and mist generated, and a process of determining the probability of smoke and mist generation.

[0145] The process by the determination unit 202 to determine the amount of smoke and mist generated is similar to the process by the determination unit 102 in the first embodiment.

[0146] An example of a process for determining the probability (ratio) of smoke and mist generation by the determination unit 202 will be described. The determination unit 202 calculates a probability P1 of smoke generation and a probability P2 of mist generation (representative values ​​of the probability of smoke and mist generation) based on the dynamic range in the same manner as the determination unit 102 of the first embodiment.

[0147] Here, the determination unit 202 corrects the probability P1 of smoke generation and the probability P2 of mist generation based on the device-used information. When the device-used information indicates that an electric scalpel is in use and that an ultrasonic coagulating incision device is not in use, the determination unit 202 updates the probability P1 of smoke generation by adding a predetermined probability value to the probability P1 of smoke generation. The determination unit 202 also updates the probability P2 of mist generation by subtracting a predetermined probability value from the probability P2 of mist generation.

[0148] When the device usage information indicates that the electric scalpel is not in use and that the ultrasonic coagulating and incising device is in use, the determination unit 202 updates the probability P1 of smoke generation by subtracting a predetermined probability value from the probability P1 of smoke generation. Also, the determination unit 202 updates the probability P2 of mist generation by adding a predetermined probability value to the probability P2 of mist generation.

[0149] If the electric scalpel and ultrasonic coagulating and incising device are in use, the determination unit 202 sets the probability P1 of smoke generation and the probability P2 of mist generation as the generation probabilities based on the device-in-use information.

[0150] The determination unit 202 outputs the determination result to the generation unit 205. The determination result includes information on whether or not smoke or mist is generated in the input image. If smoke or mist is generated, the determination result further includes the amount of smoke and mist generated and the probability P1, P2 of smoke and mist generation.

[0151] ((Smoke removal processing unit 203)) The smoke removal processing unit 203 is a processing unit that generates a smoke-removed image by removing smoke from an input image. The smoke removal processing unit 203 outputs the smoke-removed image to the generation unit 205. The description of the smoke removal processing unit 203 is similar to the processing of the smoke removal processing unit 103 in the first embodiment.

[0152] ((Mist removal processing unit 204)) The mist removal processing unit 204 is a processing unit that generates a mist-removed image by removing mist from an input image. The mist removal processing unit 204 outputs the mist-removed image to the generation unit 205. The description of the mist removal processing unit 204 is similar to the processing of the mist removal processing unit 104 in the first embodiment.

[0153] ((Generation unit 205)) The generating unit 205 is a processing unit that generates an output image based on the determination result by the determining unit 202 and the input image. The generating unit 205 outputs information on the output image to the display device 5155. The description of the generating unit 205 is similar to the processing of the generating unit 105 in the first embodiment.

[0154] <<3.3. Effects of the information processing device according to the second embodiment>> According to the information processing device 200 according to the second embodiment of the present disclosure, it is determined whether or not smoke or mist is included in the input image based on the device information and the input information, and the probability of smoke and mist occurrence is corrected based on the device information. This makes it possible to improve the accuracy of determining whether or not smoke or mist is included in the input image. In addition, by correcting the probability of smoke and mist occurrence based on the device information, it is possible to blend the smoke-excluded image and the mist-excluded image with a more appropriate blend ratio α.

[0155] <4. Third embodiment> <<4.1. System configuration according to the third embodiment>> Next, a third embodiment of the present disclosure will be described in detail. The system configuration according to the third embodiment of the present disclosure is similar to the system configuration according to the second embodiment of the present disclosure described in FIG. 11. In the following description, the information processing device according to the third embodiment will be referred to as an information processing device 300. Although not shown in the drawings, the imaging device 10, the device-in-use monitoring device 60, the display device 5155, and the information processing device 300 are connected to each other via a network 20.

[0156] <<4.2. Configuration of the information processing device according to the third embodiment>> Next, an information processing device 300 according to a third embodiment of the present disclosure will be described in detail. Fig. 13 is a diagram showing a configuration example of an information processing device according to the third embodiment of the present disclosure. As shown in Fig. 13, the information processing device 300 has a storage unit 301, a determination unit 302, a parameter generation unit 303, and a smoke removal processing unit 304.

[0157] Every time the information processing device 300 receives an input image from the imaging device 10, it inputs the input image to the determination unit 302 and the smoke removal processing unit 203. Every time the information processing device 300 receives device-used information from the device-used monitoring device 60, it inputs device-used information to the determination unit 302. Although not shown in FIG. 13, the information processing device 300 is assumed to have a communication unit that communicates information with the imaging device 10, the device-used monitoring device 60, and the display device 5155 via the network 20.

[0158] It should be noted that the information processing device 300 does not distinguish between smoke and mist, and treats either smoke or mist as smoke.

[0159] ((Storage unit 301)) The storage unit 301 is a storage device that stores information on the latest output image generated by the smoke removal processing unit 304. The output image is an image in which no smoke is generated (or from which smoke has been removed). The output image stored in the storage unit 301 is updated every time a new output image is output from the smoke removal processing unit 304.

[0160] The storage unit 301 corresponds to a semiconductor memory element such as a RAM, a ROM, or a flash memory, or a storage device such as a HDD, etc. The storage unit 301 may be either a volatile memory or a non-volatile memory, or may use both.

[0161] ((Judgment unit 302)) The determination unit 302 is a processing unit that determines whether or not smoke is included in the input image based on the device information and the input image. If the determination unit 302 determines that smoke is included, it determines the amount of smoke generated.

[0162] An example of a process for determining whether or not smoke is included in an input image by the determination unit 302 will be described below. For example, the determination unit 302 determines that smoke is included in the input image when the device information includes information indicating that an electric scalpel or an ultrasonic coagulating and cutting device is being used.

[0163] The determination unit 302 may determine that the input image contains smoke based on the input image and may determine that the input image contains smoke when the device information includes information indicating that an electric scalpel or an ultrasonic coagulating and cutting device is used. The process by the determination unit 202 to determine that the input image contains smoke (smoke or mist) based on the input image is similar to the process by the determination unit 102 in the first embodiment.

[0164] If the determining unit 302 determines that smoke is contained in the input image, it executes a process of determining the amount of smoke generated and a process of identifying the area where the smoke is generated.

[0165] For example, the determining unit 302 divides the input image into a plurality of blocks, and calculates the change in luminance and saturation over time for each block.

[0166] The determination unit 302 determines the block BO of the output image. ij and the luminance of the block BI of the input image. ij The difference between the brightness of the BO and the brightness of the ij BI indicates the block in the i-th row and j-th column of the divided blocks of the output image. ij indicates the block in the i-th row and j-th column of the divided blocks of the input image.

[0167] The determination unit 302 determines the block BO of the output image. ij and the saturation of the block BI of the input image ij The difference between the saturation of the first color and the saturation of the second color is calculated as the change in saturation over time.

[0168] The determination unit 302 compares each block of the output image with each block of the input image, and calculates the change in luminance and saturation over time for each block. The determination unit 102 identifies, among the blocks of the input image, blocks whose change in luminance and saturation over time is equal to or greater than a threshold value as a region where smoke is generated.

[0169] The determination unit 302 determines the amount of smoke generated based on a "generation amount determination table (not shown)" that defines the relationship between the ratio of blocks in the smoke generation area among all blocks of the input image, the time change of brightness and saturation, and the amount of smoke generated. The information in the generation amount determination table is set in advance.

[0170] The determination unit 302 outputs the determination result to the parameter generation unit 303. The determination result includes information on whether or not smoke is occurring in the input image. If smoke is occurring, the determination result further includes information on the amount of smoke occurring and the area where the smoke is occurring.

[0171] ((Parameter generating unit 303)) The parameter generating unit 303 is a processing unit that generates parameters for the smoke removal process based on the determination result of the determining unit 302. The parameters generated by the parameter generating unit 303 include the on / off timing of the smoke removal process, the strength level of the smoke removal process, and information on the area to be targeted by the smoke removal process.

[0172] An example of the processing of the parameter generating unit 303 while the determination result includes information indicating that no smoke is occurring in the input image will be described below. The parameter generating unit 303 acquires the determination result of the determining unit 302, and while the input image includes information indicating that no smoke is occurring, generates parameters that set the smoke removal processing to "OFF" and outputs the parameters to the smoke removal processing unit 304.

[0173] It should be noted that while the input image contains information indicating that no smoke is occurring, the parameter generating unit 303 does not set the intensity level of the smoke removal process and information on the area to be targeted for the smoke removal process as parameters.

[0174] Next, an example of the processing of the parameter generating unit 303 while the judgment result includes information that smoke is occurring in the input image will be described. The parameter generating unit 303 acquires the judgment result of the judging unit 302, and generates parameters that set the smoke removal processing to "ON" while the input image includes information that smoke is occurring.

[0175] The parameter generating unit 303 determines the intensity level based on an "intensity level determination table (not shown)" that defines the relationship between the amount of smoke generated, which is included in the determination result, and the intensity level. The intensity level determination table is set in advance so that the greater the amount of smoke generated, the higher the intensity level. The parameter generating unit 303 sets the intensity level as a parameter.

[0176] Furthermore, the parameter generating unit 303 sets information on the area where smoke is generated, which is included in the determination result, as information on the area that is the target of the smoke removal process, in the parameters.

[0177] As described above, the parameter generating unit 303 sets the smoke removal process to "on" and outputs parameters in which the intensity level and information on the area to be subject to the smoke removal process are set to the smoke removal processing unit 304.

[0178] ((Smoke removal processing unit 304)) The smoke removal processing unit 304 is a processing unit that generates a smoke-removed image by removing smoke from an input image based on parameters. In the third embodiment, the smoke-removed image corresponds to an output image. The smoke removal processing unit 304 outputs the smoke-removed image (output image) to the display device 5155. In addition, the smoke removal processing unit 304 registers the smoke-removed image in the storage unit 301.

[0179] If the smoke removal processing is set to "off" in the parameters, the smoke removal processing unit 304 does not perform the smoke removal processing and outputs the input image as is.

[0180] If the smoke removal process is set to "on" in the parameters, the smoke removal processor 304 executes the following smoke removal process.

[0181] The smoke removal processor 304 divides the input image into a plurality of blocks, compares the information of the area to be subjected to the smoke removal process with each block, and selects the block to be subjected to the smoke removal process. The smoke removal processor 304 executes a process equivalent to the deterioration estimation unit 31 and a process equivalent to the deterioration correction unit 32 for the selected block in the same manner as the smoke removal processor 304 in the first embodiment.

[0182] Here, when converting pixel values ​​of the input image to pixel values ​​of the smoke-removed image, the smoke removal processor 304 imposes a limit on the width of contrast that is allowed to change, depending on the intensity level. When the change in pixel value of the smoke-removed image relative to the pixel value of the input image exceeds the allowable contrast width, the smoke removal processor 304 adjusts the pixel value of the pixel of the smoke-removed image so that it falls within the allowable contrast width. The relationship between the intensity level and the allowable contrast width is set in advance in a "contrast specification table (not shown)". In the contrast specification table, the higher the intensity level, the wider the contrast width.

[0183] <<4.3. Effects of the information processing device according to the third embodiment>> According to the information processing device 300 according to the third embodiment of the present disclosure, parameters for performing the smoke removal process are generated based on the device information and the input information. Since the parameters are optimized by the parameter generating unit 303, the smoke removal process is performed using the optimized parameters, thereby generating an output image in which smoke has been appropriately removed from the input image.

[0184] <5. Fourth embodiment> <<5.1. System configuration according to the fourth embodiment>> Next, a fourth embodiment of the present disclosure will be described in detail. The system configuration according to the fourth embodiment of the present disclosure is similar to the system configuration according to the first embodiment of the present disclosure described in FIG. 2. In the following description, the information processing device according to the fourth embodiment will be referred to as an information processing device 400. Although not shown in the figure, the imaging device 10, the display device 5155, and the information processing device 400 are connected to each other via a network 20.

[0185] It should be noted that the information processing device 400 does not distinguish between smoke and mist, and treats either smoke or mist as smoke.

[0186] <<5.2. Configuration of the information processing device according to the fourth embodiment>> Next, an information processing device 400 according to a fourth embodiment of the present disclosure will be described in detail. Fig. 14 is a diagram showing an example of the configuration of an information processing device according to the fourth embodiment of the present disclosure. As shown in Fig. 14, the information processing device 400 has a first smoke removal processing unit 401, a subtraction unit 402, a determination unit 403, a parameter generation unit 404, and a second smoke removal processing unit 405.

[0187] Every time the information processing device 400 receives an input image from the imaging device 10, it inputs the input image to the first smoke removal processing unit 401 and the subtraction unit 402. Although not shown in Fig. 14, the information processing device 400 has a communication unit that performs information communication with the imaging device 10 and the display device 5155 via the network 20.

[0188] ((First smoke removal processing unit 401)) The first smoke removal processing unit 401 is a processing unit that generates a smoke-removed image by removing smoke from an input image based on preset initial parameters. The first smoke removal processing unit 401 outputs the smoke-removed image to a subtraction unit 402. The process of the first smoke removal processing unit 401 generating the smoke-removed image using parameters (initial parameters) is similar to the process of the smoke removal processing unit 304 according to the third embodiment.

[0189] ((Subtraction unit 402)) The subtraction unit 402 is a processing unit that generates a difference image between an input image and a smoke-removed image. For example, the subtraction unit 402 generates the difference image by subtracting the smoke-removed image from the input image. The subtraction unit 402 outputs information on the difference image to the determination unit 403.

[0190] ((Judgment unit 403)) The determination unit 403 is a processing unit that determines whether or not smoke is included in the input image based on the difference image. If it is determined that smoke is included, the determination unit 403 determines the amount of smoke generated.

[0191] An example of a process in which the determination unit 403 determines whether or not smoke is included in an input image will be described. For example, the determination unit 403 sums up the pixel values ​​of each pixel of the difference image, and if the summed pixel value is equal to or greater than a threshold value Th1, determines that smoke is included in the input image. The threshold value Th1 is assumed to be set in advance.

[0192] If the determining unit 403 determines that smoke is contained in the input image, it executes a process of determining the amount of smoke generated and a process of identifying the area where the smoke is generated.

[0193] For example, the determination unit 403 divides the difference image into a plurality of blocks, calculates the sum of pixel values ​​for each block, and identifies a block having a sum of pixel values ​​equal to or greater than a threshold value Th2 as a smoke generation region.

[0194] The determination unit 403 determines the amount of smoke generated based on a "generation amount determination table (not shown)" that defines the relationship between the ratio of blocks in the smoke generation area among all blocks of the input image, the total pixel value of the block, and the amount of smoke generated. The information in the generation amount determination table is assumed to be set in advance.

[0195] The determination unit 403 outputs the determination result to the parameter generation unit 404. The determination result includes information on whether or not smoke is occurring in the input image. If smoke is occurring, the determination result further includes information on the amount of smoke occurring and the area where the smoke is occurring.

[0196] ((Parameter generating unit 404)) The parameter generating unit 404 is a processing unit that generates parameters for the smoke removal process based on the determination result of the determining unit 403. The parameters generated by the parameter generating unit 404 include the on / off timing of the smoke removal process, the intensity level of the smoke removal process, and information on the area that is the target of the smoke removal process. The process by which the parameter generating unit 404 generates parameters is similar to that of the parameter generating unit 303 described in the fourth embodiment. The parameter generating unit 404 outputs the parameters to the second smoke removal processing unit 405.

[0197] ((Second smoke removal processing unit 405)) The second smoke removal processing unit 405 is a processing unit that generates a smoke-removed image by removing smoke from an input image based on parameters. The second smoke removal processing unit 405 outputs the smoke-removed image (output image) to the display device 5155. The process of the second smoke removal processing unit 405 generating the smoke-removed image using parameters is similar to the process of the smoke removal processing unit 304 according to the third embodiment.

[0198] <<5.3. Effects of the information processing device according to the fourth embodiment>> According to the information processing device 400 according to the fourth embodiment of the present disclosure, a smoke-removed image is first generated using initial parameters, a difference image between the input image and the smoke-removed image is generated, and the amount and area of ​​smoke generated is determined based on the difference image. The information processing device 400 can optimize parameters for the smoke removal process by using the determination result, and can generate an output image in which smoke has been appropriately removed from the input image by executing the smoke removal process using the parameters.

[0199] <6. Fifth embodiment> <<6.1. System configuration according to the fifth embodiment>> Next, a fifth embodiment of the present disclosure will be described in detail. Fig. 15 is a diagram showing an example of a system configuration according to the fifth embodiment of the present disclosure. As shown in Fig. 15, this system includes an imaging device 10, a device-in-use monitoring device 60, a display device 5155, an input device 5161, and an information processing device 500. The imaging device 10, the device-in-use monitoring device 60, the display device 5155, the input device 5161, and the information processing device 500 are connected to each other via a network 20.

[0200] The image capture device 10, the device-usage monitoring device 60, and the display device 5155 are similar to those described in FIG.

[0201] The input device 5161 is an input interface for the endoscopic surgery system 5113. For example, a user operates the input device 5161 to specify an area from which smoke is to be removed. Information on the area from which smoke is to be removed specified by the user is referred to as "specified information." The input device 5161 transmits the specified information to the information processing device 500 via the network 20.

[0202] The input device 5161 may also have a camera and detect the user's gaze position. Sensing information including the information on the user's gaze position is transmitted to the information processing device 500 via the network 20.

[0203] <<6.2. Configuration of information processing device according to fifth embodiment>> Next, an information processing device 500 according to a fifth embodiment of the present disclosure will be described in detail. Fig. 16 is a diagram showing a configuration example of an information processing device according to a fifth embodiment of the present disclosure. As shown in Fig. 16, the information processing device 500 has a storage unit 501, a determination unit 502, a parameter generation unit 503, and a smoke removal processing unit 504.

[0204] Each time the information processing device 500 receives an input image from the imaging device 10, it inputs the input image to the determination unit 502, the parameter generation unit 503, and the smoke removal processing unit 504. Each time the information processing device 300 receives device-used information from the device-used monitoring device 60, it inputs the device-used information to the determination unit 502. Each time the information processing device 500 receives designation information and sensing information from the input device 5161, it outputs the designation information and sensing information to the parameter generation unit 503. Although not shown in FIG. 16, the information processing device 500 has a communication unit that communicates information with the imaging device 10, the device-used monitoring device 60, the input device 5161, and the display device 5155 via the network 20.

[0205] It should be noted that the information processing device 500 does not distinguish between smoke and mist, and treats either smoke or mist as smoke.

[0206] ((Storage unit 501)) The storage unit 501 is a storage device that stores information on the latest output image generated by the smoke removal processing unit 504. The output image is an image in which no smoke is generated (or from which smoke has been removed). The output image stored in the storage unit 501 is updated every time a new output image is output from the smoke removal processing unit 504.

[0207] The storage unit 501 corresponds to a semiconductor memory element such as a RAM, a ROM, or a flash memory, or a storage device such as a HDD, etc. The storage unit 501 may be either a volatile memory or a non-volatile memory, or may use both.

[0208] ((Judgment unit 502)) The determination unit 502 is a processing unit that determines whether or not smoke is included in the input image based on the device information and the input image. If the determination unit 502 determines that smoke is included, it determines the amount of smoke generated. The process of the determination unit 502 is similar to the process of the determination unit 302 described in the third embodiment.

[0209] The determination unit 502 outputs the determination result to the parameter generation unit 503. The determination result includes information on whether or not smoke is occurring in the input image. If smoke is occurring, the determination result further includes information on the amount of smoke occurring and the area where the smoke is occurring.

[0210] ((Parameter generating unit 503)) The parameter generating unit 503 is a processing unit that generates parameters for the smoke removal process based on the determination result, specification information, and sensing information of the determining unit 502. The parameters generated by the parameter generating unit 503 include the on / off timing of the smoke removal process, the strength level of the smoke removal process, and information on the area to be targeted for the smoke removal process.

[0211] An example of the processing of the parameter generating unit 503 while the determination result includes information indicating that no smoke is occurring in the input image will be described. The parameter generating unit 503 acquires the determination result of the determining unit 502, and while the input image includes information indicating that no smoke is occurring, generates parameters that set the smoke removal processing to "OFF" and outputs the parameters to the smoke removal processing unit 504.

[0212] Note that while the input image contains information indicating that no smoke is occurring, the parameter generating unit 503 does not set the intensity level of the smoke removal process and information on the area to be targeted for the smoke removal process as parameters.

[0213] Next, an example of the processing of the parameter generating unit 503 while the judgment result includes information that smoke is occurring in the input image will be described. The parameter generating unit 503 acquires the judgment result of the judging unit 403, and generates parameters that set the smoke removal processing to "ON" while the input image includes information that smoke is occurring.

[0214] The parameter generating unit 503 determines the intensity level based on an "intensity level determination table (not shown)" that defines the relationship between the amount of smoke generated, which is included in the determination result, and the intensity level. The intensity level determination table is set in advance so that the greater the amount of smoke generated, the higher the intensity level. The parameter generating unit 503 sets the intensity level as a parameter.

[0215] Here, an example of the process in which the parameter generating unit 503 specifies an area to be the target of the smoke removal process will be described. The parameter generating unit 503 specifies an area that is a smoke generation area included in the determination result and is also a part of a preset input image as an area to be the target of the smoke removal process. In the following description, the part of the input image that is preset will be referred to as a "region of interest."

[0216] The parameter generating unit 503 may set the region of interest in any way. The parameter generating unit 503 may set the region of interest in the center of the input image. Furthermore, when the parameter generating unit 503 receives designation information, it sets the region of interest based on the designation information. When the parameter generating unit 503 receives sensing information, it sets the region of interest based on the user's gaze position.

[0217] The parameter generating unit 503 may specify the positions of organs and surgical tools based on the input image, and set a region of interest based on the specified positions of the organs and surgical tools. The parameter generating unit 503 may specify the positions of the organs and surgical tools using any conventional technology. For example, the parameter generating unit 503 extracts edges from the input image, and performs matching using a template that defines the shape of a specific organ or surgical tool, to specify the positions of the organs and surgical tools.

[0218] By executing the above processing, the parameter generation unit 503 sets parameters such as the on / off timing of the smoke removal process, the intensity level of the smoke removal process, and information on the area to be targeted by the smoke removal process, and outputs them to the smoke removal processing unit 504.

[0219] ((Smoke removal processing unit 504)) The smoke removal processing unit 504 is a processing unit that generates a smoke-removed image by removing smoke from an input image based on the parameters. The smoke removal processing unit 504 outputs the smoke-removed image (output image) to the display device 5155. The process of the smoke removal processing unit 504 generating the smoke-removed image using the parameters is similar to the process of the smoke removal processing unit 304 according to the third embodiment.

[0220] The region that is the target of the smoke removal process is the region where the smoke is generated and is included in the determination result, and is also the region that is the attention region. Fig. 17 is a diagram showing an example of an output image generated by the smoke removal process according to the fifth embodiment of the present disclosure. As shown in Fig. 17, the output image 70 includes a region 70a where the smoke removal process has been performed and a region 70b where the smoke removal process has not been performed.

[0221] <<6.3. Effects of the information processing device according to the fifth embodiment>> According to the information processing device 500 according to the fifth embodiment of the present disclosure, the area to be subjected to the smoke removal process is limited to the area that is the smoke generation area included in the determination result and is the area of ​​interest. As a result, the smoke removal process is not performed on areas other than the area of ​​interest, so that smoke can be removed from important parts during surgery and the user can easily check whether the smoke removal process is effective or not. In addition, it is possible to prevent the erroneous removal of things other than smoke.

[0222] <7. Sixth embodiment> <<7.1. System configuration according to the sixth embodiment>> Next, a sixth embodiment of the present disclosure will be described in detail. The system configuration according to the sixth embodiment of the present disclosure is similar to the system configuration according to the first embodiment of the present disclosure described in FIG. 2. In the following description, the information processing device according to the sixth embodiment will be referred to as an information processing device 600. Although not shown in the drawings, the imaging device 10, the display device 5155, and the information processing device 400 are connected to each other via a network 20.

[0223] It should be noted that the information processing device 600 does not distinguish between smoke and mist, and treats either smoke or mist as smoke.

[0224] <<7.2. System configuration according to the sixth embodiment>> Next, an information processing device 600 according to a sixth embodiment of the present disclosure will be described in detail. Fig. 18 is a diagram showing an example of the configuration of an information processing device according to the sixth embodiment of the present disclosure. As shown in Fig. 18, the information processing device 600 has a smoke removal processing unit 601, a subtraction unit 602, a determination unit 603, a parameter generation unit 604, and a superposition unit 605.

[0225] Every time the information processing device 600 receives an input image from the imaging device 10, it inputs the input image to the smoke removal processing unit 601 and the subtraction unit 402. Although not shown in Fig. 18, the information processing device 600 has a communication unit that performs information communication with the imaging device 10 and the display device 5155 via the network 20.

[0226] ((Smoke removal processing unit 601)) The smoke removal processing unit 601 is a processing unit that generates a smoke-removed image by removing smoke from an input image based on parameters acquired from a parameter generation unit 604. The smoke removal processing unit 601 outputs the smoke-removed image to a subtraction unit 602 and a superposition unit 605. The process by which the smoke removal processing unit 601 generates a smoke-removed image using parameters is similar to the process of the smoke removal processing unit 304 according to the third embodiment.

[0227] (Subtraction unit 602) The subtraction unit 602 is a processing unit that generates a difference image between the input image and the smoke-removed image. For example, the subtraction unit 602 generates the difference image by subtracting the smoke-removed image from the input image. The subtraction unit 602 outputs information on the difference image to the determination unit 603.

[0228] ((Judgment unit 603)) The determination unit 603 is a processing unit that determines whether or not smoke is included in the input image based on the difference image. If the determination unit 603 determines that smoke is included, it determines the amount of smoke generated. The process of the determination unit 603 is similar to the process of the determination unit 403 according to the fourth embodiment.

[0229] The determination unit 603 outputs the determination result to the parameter generation unit 604. The determination result includes information on whether or not smoke is occurring in the input image. If smoke is occurring, the determination result further includes information on the amount of smoke occurring and the area where the smoke is occurring.

[0230] ((Parameter generating unit 604)) The parameter generating unit 604 is a processing unit that generates parameters for the smoke removal process based on the determination result of the determining unit 603. The parameters generated by the parameter generating unit 604 include the on / off timing of the smoke removal process, the strength level of the smoke removal process, and information on the area that is the target of the smoke removal process. The process by which the parameter generating unit 604 generates parameters is similar to that of the parameter generating unit 303 described in the fourth embodiment. The parameter generating unit 604 outputs the parameters to the smoke removal processing unit 601.

[0231] In addition, the parameter generating unit 604 outputs information on the strength level of the smoking removal process to the superimposing unit 605.

[0232] ((Superimposing unit 605)) The superimposing unit 605 is a processing unit that superimposes information on the intensity level of the smoke removal process on the output image. FIG. 19 is a diagram for explaining the processing of the superimposing unit according to the sixth embodiment of the present disclosure. In the example shown in FIG. 19, information 71a indicating "intensity level: 80" is superimposed on the output image 71. The superimposing unit 605 outputs the output image resulting from the processing to the display device 5155.

[0233] <<7.3. Effects of the information processing device according to the sixth embodiment>> According to the information processing device 600 according to the sixth embodiment of the present disclosure, information on the intensity level of the smoke removal process is superimposed on the output image. This allows the user to determine whether the smoke removal process is effective or not. In addition, by displaying the numerical value of the intensity level, the user can determine the amount of smoke generated and the effectiveness of the smoke removal.

[0234] <<8. Hardware Configuration>> The information processing device according to each embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 20, for example. The information processing device 100 according to the first embodiment will be described below as an example. Fig. 20 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0235] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0236] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0237] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0238] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0239] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0240] For example, when the computer 1000 functions as the information processing device 100 according to the first embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded on the RAM 1200 to realize functions of the determination unit 102, the smoke removal processing unit 103, the mist removal processing unit 104, the generation unit 105, and the like. The HDD 1400 stores the generation program according to the present disclosure and data in the storage unit 101. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0241] <9. Conclusion> The information processing device has a generation unit. The generation unit acquires an input image, which is an image related to surgery, and generates an output image based on whether or not the input image contains a substance generated during surgery. The information processing device also has a determination unit. The determination unit determines whether or not the input image contains smoke or mist. The determination unit further determines the amount of smoke or mist generated based on the input image. The determination unit further determines the ratio of smoke to mist based on the input image. The generation unit is characterized by excluding the influence of smoke or mist from the entire output image. As a result, even if smoke or mist is generated during endoscopic surgery and visibility becomes poor, by using the information processing device, it is possible to ensure a clearer view regardless of the amount of smoke or mist generated.

[0242] The determination unit determines whether or not the input image contains smoke or mist by further using the type and operation status of the electronic device connected to the information processing device, thereby improving the accuracy of determining whether or not the input image contains smoke or mist.

[0243] The information processing device further includes a smoke removal processing unit that generates a smoke-removed image by removing smoke from the input image. The generation unit generates the output image using the determination result, the input image, and the smoke-removed image. This makes it possible to generate a smoke-removed image by removing smoke, and to generate an output image in which no smoke is generated using the smoke-removed image.

[0244] The smoke removal processing unit estimates the deterioration of the input image based on the output image and the input image, and generates the smoke-removed image based on the estimation result, thereby making it possible to appropriately remove smoke contained in the input image.

[0245] The information processing device further includes a superimposing unit that superimposes information about the smoke removed by the smoke removal processing unit on the output image. Here, the information about the smoke may be any information related to the reduced smoke, such as whether or not the smoke reduction process has been performed, the degree of smoke reduction, and the strength of the smoke reduction process. This allows the user to determine whether the smoke removal process is effective or not. Also, by displaying a numerical value of the strength level, the user can determine how much smoke is generated and how effective the smoke removal is.

[0246] The information processing device further includes a subtraction unit that generates a difference image between the input image and the smoke-removed image, and the determination unit identifies an amount of smoke generated based on the difference image, and generates information about the smoke removed by the smoke removal processing unit based on the amount of smoke generated. The information processing device further includes a parameter generation unit that generates parameters to be used in the smoke removal process based on a determination result of the determination unit, and the smoke removal processing unit generates a smoke-removed image in which smoke has been removed from the input image based on the parameters. The parameter generation unit generates parameters including the timing of starting or ending the smoke removal process, the strength of the smoke removal process, and the target area of ​​the smoke removal process based on the determination result of the determination unit. In this way, by using the difference image, the parameters of the smoke removal process can be optimized, and by executing the smoke removal process using such parameters, an output image in which smoke has been appropriately removed from the input image can be generated.

[0247] The information processing device further includes a mist removal processing unit that generates a mist-removed image in which mist has been removed from the input image, and the generation unit generates the output image using the determination result, the input image, and the mist-removed image. The mist removal processing unit identifies a mist generation area based on the input image, generates a corrected image in which the mist generation area is corrected using information about an area around the mist generation area, estimates deterioration of the corrected image based on the corrected image and the output image, and generates the mist-removed image based on the estimated result. In this way, by adjusting the contrast of the input image in two stages, even mottled mist, which has different characteristics from smoke, can be appropriately removed from the input image.

[0248] The generation unit generates the output image by combining the input image, the smoke-removed image, and the mist-removed image based on the generation amount and the ratio of smoke to mist. This makes it possible to combine the smoke-removed image and the mist-removed image using a blend ratio α based on the generation probability of smoke or mist, and to appropriately remove the smoke and mist contained in the input image.

[0249] The generating unit is characterized by excluding the influence of smoke or mist from a partial area of ​​the output image. As a result, the smoke removal process is not performed on areas other than the area of ​​interest, so that smoke can be removed from important areas during surgery, and the user can easily check whether the smoke removal process is effective or not. In addition, it is possible to prevent the erroneous removal of things other than smoke.

[0250] The generating unit identifies the partial area based on the position of an organ or surgical tool identified from the input image, and removes the effects of smoke or mist from the partial area of ​​the output image. The generating unit identifies the partial area based on the user's viewpoint position, and removes the effects of smoke or mist from the partial area of ​​the output image. This makes it possible to clear the area that the user is paying attention to.

[0251] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0252] The present technology can also be configured as follows. (1) a generation unit that acquires an input image that is an image related to an operation, and generates an output image based on whether or not the input image contains a substance that occurs during the operation; An information processing device having the above configuration. (2) The information processing device according to (1) above, further comprising a determination unit that determines whether or not the input image contains smoke or mist. (3) The information processing device described in (2) is characterized in that the determination unit further uses the type and operating status of an electronic device connected to the information processing device to determine whether or not the input image contains smoke or mist. (4) The information processing device according to (2) or (3), wherein the determination unit further determines an amount of smoke or mist generated based on the input image. (5) The information processing device according to (2), (3) or (4), wherein the determination unit further determines a ratio of smoke to mist based on the input image. (6) The information processing device described in (4) further comprises a smoke removal processing unit that generates a smoke-removed image by removing smoke from the input image, and the generation unit generates the output image using the judgment result, the input image, and the smoke-removed image. (7) The information processing device described in (6) is characterized in that the smoke removal processing unit estimates deterioration of the input image based on the output image and the input image, and generates the smoke-removed image based on the estimated result. (8) The information processing device according to (7) above, further comprising a superimposition unit that superimposes information about the smoke removed by the smoke removal processing unit onto the output image. (9) The information processing device described in (8) further comprises a subtraction unit that generates a difference image between the input image and the smoke-removed image, and the judgment unit identifies the amount of smoke generated based on the difference image, and generates information regarding the smoke removed by the smoke removal processing unit based on the amount of smoke generated. (10) The information processing device described in (6), (7) or (8) is characterized in that it further comprises a mist removal processing unit that generates a mist-removed image in which mist has been removed from the input image, and the generation unit generates the output image using the judgment result, the input image, and the mist-removed image. (11) The information processing device described in (10) is characterized in that the mist removal processing unit identifies an area where mist is generated based on the input image, generates a corrected image in which the area where mist is generated is corrected using information on the area surrounding the area where the mist is generated, estimates deterioration of the corrected image based on the corrected image and the output image, and generates the mist removed image based on the estimated result. (12) The information processing device described in (11) is characterized in that the generation unit generates the output image by combining the input image, the smoke-removed image, and the mist-removed image based on the amount of smoke generated and the ratio of smoke to mist. (13) The information processing device described in any one of (6) to (12) is characterized in that it further comprises a parameter generation unit that generates parameters to be used in smoke removal processing based on the judgment result of the judgment unit, and the smoke removal processing unit generates a smoke-removed image in which smoke has been removed from the input image based on the parameters. (14) The information processing device described in (13) is characterized in that the parameter generation unit generates parameters including the timing of starting or ending the smoke removal process, the strength of the smoke removal process, and the target area of ​​the smoke removal process based on the judgment result of the judgment unit. (15) The information processing device according to any one of (2) to (14), wherein the generating unit removes the influence of smoke or mist from the entire output image. (16) The information processing device according to any one of (2) to (14), wherein the generating unit removes an effect of smoke or mist from a partial area of ​​the output image. (17) The information processing device described in (16) is characterized in that the generation unit identifies the partial area based on the position of an organ or surgical tool identified from the input image, and excludes the effects of smoke or mist from the partial area of ​​the output image. (18) The information processing device described in (16) is characterized in that the generation unit identifies the portion of the area based on the user's viewpoint position and excludes the effects of smoke or mist from the portion of the area of ​​the output image. (19) The computer Obtaining an input image, which is an intraoperative image; Generate an output image based on whether the input image contains intraoperatively generated material. The generation method to perform the operation. (20) Computer, a generation unit that acquires an input image that is an image related to an operation, and generates an output image based on whether or not the input image contains a substance that occurs during the operation; A generator to function as a. [Explanation of symbols]

[0253] 10. Imaging device 11 Imaging unit 12,62 Communications Department 20 Network 31,43 Deterioration Estimation Unit 32 Deterioration correction section 41 Occurrence area identification section 42 First deterioration correction unit 44 Second deterioration correction section 51 First blend ratio calculation unit 52 First blend processing section 53 Second blend ratio calculation unit 54 Second blend processing section 60 Device monitoring equipment 61 Monitoring Department 100, 200, 300, 400, 500, 600 Information processing equipment 101,201,301,501 Storage section 102,202,302,403,502,603 ​​Judgment section 103,203,304,504,601 Smoke removal processing unit 104,204 Mist removal processing unit 105,205 Generation part 303,404,503,604 Parameter Generation Unit 401 First smoke removal processing section 402 Subtraction section 405 Second smoke removal processing section 605 Overlapped part

Claims

1. A determination unit that acquires an input image that is an image related to an operation and determines whether or not the input image contains smoke or mist generated during the operation; A generator for generating an output image; having The determination unit is Further determining a ratio of the smoke to the mist based on the input image. Information processing device.

2. The information processing device according to claim 1 , wherein the determination unit determines whether or not the input image contains smoke or mist by further using a type and an operating status of an electronic device connected to the information processing device.

3. The information processing apparatus according to claim 1 , wherein the determination unit further determines an amount of smoke or mist generated based on the input image.

4. The information processing device described in claim 3, further comprising a smoke removal processing unit that generates a smoke-removed image by removing smoke from the input image, and the generation unit generates the output image using a determination result of whether or not the input image contains smoke or mist generated during the surgery, the input image, and the smoke-removed image.

5. The information processing device according to claim 4, characterized in that the smoke removal processing unit estimates deterioration of the input image based on the output image and the input image, and generates the smoke-removed image based on the estimated result.

6. 6. The information processing apparatus according to claim 5, further comprising a superimposing unit that superimposes information about the smoke removed by the smoke removal processing unit on the output image.

7. The information processing device described in claim 6, further comprising a subtraction unit that generates a difference image between the input image and the smoke-removed image, wherein the judgment unit identifies the amount of smoke generated based on the difference image, and generates information regarding the smoke removed by the smoke removal processing unit based on the amount of smoke generated.

8. The information processing device according to claim 4, further comprising a mist removal processing unit that generates a mist-removed image by removing mist from the input image, wherein the generation unit generates the output image using the judgment result, the input image, and the mist-removed image.

9. The information processing device described in claim 8, characterized in that the mist removal processing unit identifies a mist generation area based on the input image, generates a corrected image in which the mist generation area is corrected using information on the area surrounding the mist generation area, estimates deterioration of the corrected image based on the corrected image and the output image, and generates the mist removed image based on the estimated result.

10. The information processing device according to claim 9, characterized in that the generation unit generates the output image by combining the input image, the smoke-removed image, and the mist-removed image based on the amount of smoke generated and the ratio of smoke to mist.

11. The information processing device according to claim 4, further comprising a parameter generation unit that generates parameters to be used in the smoke removal processing based on the judgment result of the judgment unit, and the smoke removal processing unit generates a smoke-removed image in which smoke has been removed from the input image based on the parameters.

12. The information processing device according to claim 11, characterized in that the parameter generation unit generates parameters including the timing of starting or ending the smoke removal process, the strength of the smoke removal process, and the target area of ​​the smoke removal process based on the judgment result of the judgment unit.

13. The information processing apparatus according to claim 1 , wherein the generating unit removes the influence of smoke or mist from the entire output image.

14. The information processing apparatus according to claim 1 , wherein the generating unit removes the influence of smoke or mist from a partial area of ​​the output image.

15. The information processing device according to claim 14, characterized in that the generation unit identifies the partial area based on the position of an organ or surgical tool identified from the input image, and excludes the effects of smoke or mist from the partial area of ​​the output image.

16. The information processing apparatus according to claim 14 , wherein the generating unit specifies the partial area based on a user's viewpoint position, and removes an effect of smoke or mist from the partial area of ​​the output image.

17. The computer Obtaining an input image, which is an intraoperative image; determining whether the input image contains smoke or mist generated during surgery; Generate the output image, Further determining a ratio of the smoke to the mist based on the input image. The generation method to perform the operation.

18. Computer, A determination unit that acquires an input image that is an image related to an operation and determines whether or not the input image contains smoke or mist generated during the operation; A generator for generating an output image; having a determination unit that further determines a ratio of the smoke to the mist based on the input image; A generator to function as a.

Images