Information processing device, generation method, and generation program

Abstract

This information processing apparatus is provided with a generation unit which generates an output image by correcting a first image that is an intraoperative image on the basis of deterioration in the first image due to a substance produced during an operation, the deterioration being estimated on the basis of the first image and a second image that is an image preceding the first image.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a generation method, and a generation program. [Background technology]

[0002] In the medical field, endoscopic surgery using an endoscope is widely performed. Also, various devices related to endoscopic surgery have been developed. For example, Patent Document 1 discloses a device that, when smoke is detected from a captured endoscopic image, operates an insufflation device to remove the smoke.

[0003] Patent document 2 discloses an apparatus that, when smoke is detected from a captured endoscopic image, removes the smoke from the endoscopic image through uniform signal processing, and then controls a smoke evacuation device in accordance with the smoke detection results to remove the smoke. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 11-318909 [Patent Document 2] JP 2018-157917 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, even if the presence or absence of smoke is detected, it does not detect the amount of smoke, and depending on the amount of smoke generated, it may not be possible to sufficiently remove the smoke. Also, in Patent Document 1, because the smoke is physically removed, it takes time for the smoke to be expelled and the field of vision to become clear.

[0006] In addition, in Patent Document 2, smoke is detected from a single endoscopic image, so if an object similar in color to smoke is present in the endoscopic image, it can be difficult to distinguish between the object and the smoke, and erroneous detection of smoke can result in a degraded image after the smoke has been removed.

[0007] Therefore, the present disclosure proposes an information processing device, a generation method, and a generation program capable of generating an image in which degradation due to substances occurring during surgery has been corrected. [Means for solving the problem]

[0008] In order to solve the above problems, one form of information processing device according to the present disclosure includes a generation unit that generates an output image by correcting a first image based on degradation of the first image due to a substance occurring during surgery, which is estimated based on a first image, which is an image related to surgery, and a second image, which is an image from an earlier time than the first image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a surgical procedure to which a surgical room system using the technical concept of the present disclosure is applied. [Diagram 2] FIG. 1 is a diagram illustrating an example of a system configuration according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 13 is a diagram showing the relationship between the use time of an electronic device and the concentration of smoke. [Diagram 5] FIG. 13 is a diagram illustrating an example of a histogram. [Figure 6] FIG. 13 is a diagram illustrating an example of a cumulative histogram. [Figure 7] FIG. 2 is a diagram showing an example of an input image and an output image. [Figure 8] 1 is a flowchart illustrating a flow of a basic operation of an information processing device according to an embodiment of the present disclosure. [Figure 9] 10 is a flowchart illustrating a flow of an operation of a selection unit according to an embodiment of the present disclosure. [Figure 10] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0011] The present disclosure will be described in the following order. 1. Application Examples 2. Embodiment 2.1. System Configuration According to the Embodiment 2.2. Configuration of the information processing device according to the embodiment 2.3. Operational flow of information processing device 2.4. Effects of the information processing device according to the embodiment 3. Hardware Configuration 4. Conclusion

[0012] <1. Application Examples> An application example of the technical idea common to each embodiment of the present disclosure will be described. FIG. 1 is a diagram showing an example of the state of surgery to which a surgery room system 5100 using the technical idea according to the present disclosure is applied. A ceiling camera 5187 and a surgery room camera 5189 are provided on the ceiling of the surgery room, and can capture the hands of an operator (doctor) 5181 who is performing treatment on an affected part of a patient 5185 on a patient bed 5183, as well as the entire state of the surgery room. The ceiling camera 5187 and the surgery room camera 5189 may be provided with a magnification adjustment function, a focal length adjustment function, a shooting direction adjustment function, and the like. A light 5191 is provided on the ceiling of the surgery room, and illuminates at least the hands of the operator 5181. The light 5191 may be capable of appropriately adjusting the amount of light emitted, the wavelength (color) of the light emitted, the light irradiation direction, and the like.

[0013] The endoscopic surgery system 5113, the patient bed 5183, the ceiling camera 5187, the surgery site camera 5189, and the lighting 5191 are connected to each other via an audiovisual controller and a surgery room control device (not shown). A centralized operation panel 5111 is provided in the surgery room, and a user can appropriately operate these devices in the surgery room via the centralized operation panel 5111.

[0014] Hereinafter, a detailed description will be given of the configuration of the endoscopic surgery system 5113. As shown in the figure, the endoscopic surgery system 5113 is composed of an endoscope 5115, other surgical tools 5131, a support arm device 5141 that supports the endoscope 5115, and a cart 5151 on which various devices for endoscopic surgery are mounted.

[0015] In endoscopic surgery, instead of cutting the abdominal wall and opening the abdomen, a cylindrical opening instrument called trocar 5139a to 5139d is punctured into the abdominal wall. Then, the lens barrel 5117 of the endoscope 5115 and other surgical instruments 5131 are inserted into the body cavity of the patient 5185 from the trocar 5139a to 5139d. In the illustrated example, as the other surgical instruments 5131, a tube 5133, an energy treatment instrument 5135, and forceps 5137 are inserted into the body cavity of the patient 5185. Here, the tube 5133 may be configured to exhaust smoke generated in the body cavity to the outside of the body cavity. On the other hand, the tube 5133 may have a function of injecting gas into the body cavity to inflate the body cavity. The energy treatment instrument 5135 is a treatment instrument that performs incision and peeling of tissue, sealing of blood vessels, etc. by high-frequency current or ultrasonic vibration. However, the surgical tool 5131 shown in the figure is merely an example, and the surgical tool 5131 may be any of various surgical tools generally used in endoscopic surgery, such as a surgical implement, a retractor, etc.

[0016] An image of the area to be operated on in the body cavity of the patient 5185 photographed by the endoscope 5115 is displayed on the display device 5155. The surgeon 5181 performs treatment such as excising the affected area using the energy treatment tool 5135 and forceps 5137 while viewing the image of the area to be operated on displayed on the display device 5155 in real time. Although not shown in the figure, the tube 5133, the energy treatment tool 5135, and the forceps 5137 are supported by the surgeon 5181 or an assistant during surgery.

[0017] (Support arm device) The support arm device 5141 includes an arm portion 5145 extending from a base portion 5143. In the example shown, the arm portion 5145 is composed of joint portions 5147a, 5147b, 5147c and links 5149a, 5149b, and is driven under the control of an arm control device 5159. The arm portion 5145 supports the endoscope 5115, and the position and attitude of the endoscope 5115 are controlled. This allows the endoscope 5115 to be stably fixed in position.

[0018] (Endoscopy) The endoscope 5115 is composed of a lens barrel 5117, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 5185, and a camera head 5119 connected to a base end of the lens barrel 5117. In the illustrated example, the endoscope 5115 is configured as a so-called rigid lens barrel having a rigid lens barrel 5117, but the endoscope 5115 may be configured as a so-called flexible lens barrel having a flexible lens barrel 5117.

[0019] An opening in which an objective lens is fitted is provided at the tip of the lens barrel 5117. A light source device 5157 is connected to the endoscope 5115, and light generated by the light source device 5157 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 5117, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 5185. The endoscope 5115 may be a direct-viewing mirror, an oblique-viewing mirror, or a side-viewing mirror.

[0020] An optical system and an imaging element are provided inside the camera head 5119, and reflected light (observation light) from an observation target is collected on the imaging element by the optical system. The imaging element photoelectrically converts the observation light to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The image signal is transmitted to a camera control unit (CCU) 5153 as RAW data. The camera head 5119 is equipped with a function for adjusting the magnification and focal length by appropriately driving the optical system.

[0021] For example, in order to support stereoscopic vision (3D display), a plurality of imaging elements may be provided in the camera head 5119. In this case, a plurality of relay optical systems are provided inside the lens barrel 5117 to guide observation light to each of the plurality of imaging elements.

[0022] (Various devices mounted on the cart) The CCU 5153 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the operation of the endoscope 5115 and the display device 5155 in an integrated manner. Specifically, the CCU 5153 performs various image processing, such as development processing (demosaic processing), on the image signal received from the camera head 5119 in order to display an image based on the image signal. The CCU 5153 provides the image signal subjected to the image processing to the display device 5155. The above-mentioned audiovisual controller is also connected to the CCU 5153. The CCU 5153 also provides the image signal subjected to the image processing to the audiovisual controller 5107. The CCU 5153 transmits a control signal to the camera head 5119 to control its drive. The control signal may include information on the imaging conditions, such as magnification and focal length. The information on the imaging conditions may be input via the input device 5161 or via the above-mentioned centralized operation panel 5111.

[0023] The display device 5155 displays an image based on an image signal that has been subjected to image processing by the CCU 5153 under the control of the CCU 5153. When the endoscope 5115 is compatible with high-resolution photography such as 4K (3840 horizontal pixels x 2160 vertical pixels) or 8K (7680 horizontal pixels x 4320 vertical pixels) and / or compatible with 3D display, the display device 5155 may be capable of displaying high resolution and / or 3D display correspondingly. When compatible with high-resolution photography such as 4K or 8K, a display device 5155 with a size of 55 inches or more can be used to provide a more immersive feeling. In addition, multiple display devices 5155 with different resolutions and sizes may be provided depending on the application.

[0024] The light source device 5157 is composed of a light source such as an LED (light emitting diode) and supplies the endoscope 5115 with irradiation light when photographing the surgical site.

[0025] The arm control device 5159 is constituted by a processor such as a CPU, and operates according to a prescribed program to control the driving of the arm portion 5145 of the support arm device 5141 according to a prescribed control method.

[0026] The input device 5161 is an input interface for the endoscopic surgery system 5113. A user can input various pieces of information and instructions to the endoscopic surgery system 5113 via the input device 5161. For example, the user inputs various pieces of information related to surgery, such as patient physical information and information about the surgical procedure, via the input device 5161. In addition, for example, the user inputs, via the input device 5161, an instruction to drive the arm unit 5145, an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 5115, an instruction to drive the energy treatment tool 5135, etc.

[0027] The type of the input device 5161 is not limited, and the input device 5161 may be any of various known input devices. For example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5171, and / or a lever may be applied as the input device 5161. When a touch panel is used as the input device 5161, the touch panel may be provided on a display surface of the display device 5155.

[0028] Alternatively, the input device 5161 is a device worn by a user, such as a glasses-type wearable device or a head mounted display (HMD), and various inputs are made according to the user's gestures and gaze detected by these devices. The input device 5161 also includes a camera capable of detecting the user's movements, and various inputs are made according to the user's gestures and gaze detected from the image captured by the camera. The input device 5161 also includes a microphone capable of collecting the user's voice, and various inputs are made by voice via the microphone. In this way, the input device 5161 is configured to be able to input various information in a non-contact manner, and thus it becomes possible for a user (e.g., the surgeon 5181) belonging to a clean area to operate a device belonging to an unclean area in a non-contact manner. In addition, the user can operate the device without taking his / her hands off the surgical tool he / she is holding, which improves the user's convenience.

[0029] The treatment tool control device 5163 controls the driving of the energy treatment tool 5135 for cauterizing tissue, incising, sealing blood vessels, etc. The smoke exhaust device 5165 sends gas into the body cavity of the patient 5185 through the tube 5133 to inflate the body cavity in order to ensure the field of view of the endoscope 5115 and the working space of the surgeon. The smoke exhaust device 5165 also has a function of exhausting smoke generated in the body cavity in order to ensure the field of view of the endoscope 5115. The recorder 5167 is a device capable of recording various information related to the surgery. The printer 5169 is a device capable of printing various information related to the surgery in various formats such as text, image, and graph.

[0030] Below, the particularly characteristic configuration of the endoscopic surgery system 5113 will be described in further detail.

[0031] (Support arm device) The support arm device 5141 includes a base 5143 serving as a base, and an arm 5145 extending from the base 5143. In the illustrated example, the arm 5145 includes a plurality of joints 5147a, 5147b, and 5147c, and a plurality of links 5149a and 5149b connected by the joint 5147b. However, in FIG. 1, for the sake of simplicity, the configuration of the arm 5145 is simplified. In practice, the shapes, number, and arrangement of the joints 5147a to 5147c and the links 5149a and 5149b, as well as the directions of the rotation axes of the joints 5147a to 5147c, can be appropriately set so that the arm 5145 has a desired degree of freedom. For example, the arm 5145 can be preferably configured to have six or more degrees of freedom. This allows the endoscope 5115 to be moved freely within the movable range of the arm portion 5145, making it possible to insert the telescope tube 5117 of the endoscope 5115 into the body cavity of the patient 5185 from a desired direction.

[0032] The joints 5147a to 5147c are provided with actuators, and the joints 5147a to 5147c are configured to be rotatable around a predetermined rotation axis by driving the actuators. The drive of the actuators is controlled by the arm control device 5159, whereby the rotation angles of the joints 5147a to 5147c are controlled, and the drive of the arm 5145 is controlled. This allows control of the position and attitude of the endoscope 5115. At this time, the arm control device 5159 can control the drive of the arm 5145 by various known control methods such as force control or position control.

[0033] For example, the surgeon 5181 may appropriately input an operation via the input device 5161 (including the foot switch 5171), and the drive of the arm unit 5145 may be appropriately controlled by the arm control device 5159 in response to the operation input, thereby controlling the position and posture of the endoscope 5115. Through this control, the endoscope 5115 at the tip of the arm unit 5145 may be moved from an arbitrary position to an arbitrary position, and then fixedly supported at the position after the movement. The arm unit 5145 may be operated in a so-called master-slave manner. In this case, the arm unit 5145 may be remotely operated by a user via the input device 5161 installed at a location away from the operating room.

[0034] Furthermore, when force control is applied, the arm control device 5159 may perform so-called power assist control, in which the actuators of the joints 5147a to 5147c are driven so that the arm unit 5145 moves smoothly in response to an external force from the user. This allows the arm unit 5145 to be moved with a relatively light force when the user moves the arm unit 5145 while directly touching the arm unit 5145. This makes it possible to move the endoscope 5115 more intuitively and with a simpler operation, improving the convenience for the user.

[0035] Generally, in endoscopic surgery, the endoscope 5115 is supported by a doctor called a scopist. By contrast, by using the support arm device 5141, the position of the endoscope 5115 can be fixed more reliably without relying on human hands, so that an image of the surgical site can be obtained stably and the surgery can be performed smoothly.

[0036] The arm control device 5159 does not necessarily have to be provided on the cart 5151. Also, the arm control device 5159 does not necessarily have to be one device. For example, the arm control device 5159 may be provided on each of the joints 5147a to 5147c of the arm section 5145 of the support arm device 5141, and the drive control of the arm section 5145 may be realized by the multiple arm control devices 5159 cooperating with each other.

[0037] (Light source device) The light source device 5157 supplies the endoscope 5115 with irradiation light for photographing the surgical site. The light source device 5157 is composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. In this case, when the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the light source device 5157 can adjust the white balance of the captured image. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation target with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 5119 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0038] Furthermore, the light source device 5157 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 5119 in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner and synthesizing the images, it is possible to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0039] The light source device 5157 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, an excitation light is irradiated to a body tissue and the fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue and an excitation light corresponding to the fluorescence wavelength of the reagent is irradiated to the body tissue to obtain a fluorescent image. The light source device 5157 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0040] <2. Embodiment> <<2.1. Configuration of the system according to the embodiment>> Next, an embodiment of the present disclosure will be described in detail. Fig. 2 is a diagram showing an example of a system configuration according to an embodiment of the present disclosure. As shown in Fig. 2, this system includes an imaging device 10, a device-usage monitoring device 20, a display device 5155, and an information processing device 100. The imaging device 10, the device-usage monitoring device 20, the display device 5155, and the information processing device 100 are connected to each other via a network 30.

[0041] The imaging device 10 is a device that captures an in vivo image of an observation target inside a living body. The imaging device 10 may be, for example, an endoscope 5115 as described in FIG. 1. The imaging device 10 includes an imaging unit 11 and a communication unit 12.

[0042] The imaging unit 11 has a function of capturing an in vivo image of an observation subject in the living body. The imaging unit 11 according to this embodiment includes an imaging element such as a CCD (Charge Coupled device) or a CMOS (Complementary MOS). The imaging unit 11 captures an in vivo image at a predetermined frame rate (FPS: Frames Per Second).

[0043] Here, the in-vivo image according to the present embodiment broadly includes images acquired from a biological perspective for clinical, medical, and experimental purposes (biological imaging), and the imaging subject is not limited to a human being.

[0044] The communication unit 12 has a function of performing information communication with the information processing device 100 via the network 20. For example, the communication unit 12 transmits each in-vivo image captured by the imaging unit 11 to the information processing device 100 in chronological order.

[0045] The device-in-use monitoring device 20 is connected to an electric scalpel, an ultrasonic coagulating and cutting device (not shown), and monitors whether the electric scalpel or the ultrasonic coagulating and cutting device is being used. The device-in-use monitoring device 20 may be, for example, the treatment tool control device 5163 described in FIG. 1. The device-in-use monitoring device 20 has a monitoring unit 21 and a communication unit 22.

[0046] The monitoring unit 21 is a processing unit that monitors the usage status of the electric scalpel and the ultrasonic coagulating and cutting device. For example, when the monitoring unit 21 receives a control signal for starting usage from a start button of the electric scalpel or the ultrasonic coagulating and cutting device, the monitoring unit 21 determines that the electric scalpel or the ultrasonic coagulating and cutting device is in use. The monitoring unit 21 generates usage device information. The usage device information includes information on whether the electric scalpel is in use or not, and information on whether the ultrasonic coagulating and cutting device is in use or not.

[0047] For example, an electric scalpel uses heat generated by a high-frequency current to stop bleeding or make an incision on an affected area of ​​a patient 5185. An electric scalpel has a characteristic that it easily generates smoke because it burns the treated area.

[0048] The ultrasonic coagulating and cutting device uses friction caused by ultrasonic vibration to coagulate and cut the affected area of ​​the patient 5185. The ultrasonic coagulating and cutting device has a feature that mist is easily generated by ultrasonic vibration.

[0049] The information processing device 100 receives each in-vivo image in chronological order from the imaging device 10. In the following description, the in-vivo image received from the imaging device 10 is referred to as an "input image." The information processing device 100 estimates the degradation of the input image due to substances generated during surgery based on an image not including smoke and the input image. The information processing device 100 generates an output image based on the estimation result of the degradation of the input image and the input image. For example, substances generated during surgery are smoke and mist. The input image is also called a "medical image" or an "intraoperative image."

[0050] The information processing device 100 transmits the output image to the display device 5155. As described below, when the input image includes smoke, the information processing device 100 generates an output image from which the smoke has been removed. The information processing device 100 may be, for example, the CCU 5153 as described in FIG. 1.

[0051] The display device 5155 receives an output image from the information processing device 100 and displays the received output image. The rest of the description of the display device 5155 is the same as the description of the display device 5155 in FIG.

[0052] <<2.2. Configuration of the information processing device according to the embodiment>> Next, the information processing device 100 according to the first embodiment of the present disclosure will be described in detail. Fig. 3 is a diagram showing a configuration example of the information processing device according to the embodiment of the present disclosure. As shown in Fig. 3, the information processing device 100 has memories 101a and 101b, a selection unit 102, a histogram conversion unit 103, a motion compensation unit 104, a degradation estimation unit 105, and a generation unit 106. Although not shown in Fig. 3, the information processing device 100 has a communication unit that communicates information with the imaging device 10, the device-in-use monitoring device 20, and the display device 5155 via the network 30.

[0053] ((Memory 101a)) The memory 101a is a storage device that stores input images received from the imaging device 10 in chronological order. The input image is an example of a "first image." For example, the input images in the memory 101a are assigned frame numbers in ascending order, and each input image up to frame number n is stored. The memory 101a corresponds to a storage device such as a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a HDD (Hard Disk Drive). The memory 101a may be either a volatile memory or a non-volatile memory, or both may be used.

[0054] ((Memory 101b)) The memory 101b is a storage device that stores the pre-degraded image selected by the selection unit 102. When the memory 101b stores a past pre-degraded image, the memory 101b updates the past pre-degraded image with the newly selected pre-degraded image. The memory 101b corresponds to a storage device such as a semiconductor memory element such as a RAM or a flash memory, or a HDD. The memory 101b may be either a volatile memory or a non-volatile memory, or may use both.

[0055] ((Selection unit 102)) The selection unit 102 is a processing unit that selects an input image stored in the memory 101a or an output image output from the generation unit 106 based on device-used information received from the device-used monitoring apparatus 20. The selection unit 102 stores the selected image in the memory 101b. The image selected by the selection unit 102 is a "pre-degradation image" in which no smoke or mist occurs in the image. The pre-degradation image is an example of a "second image."

[0056] Based on the device usage information, when either the electric scalpel or the ultrasonic coagulation and cutting device is in use, the selection unit 102 selects the output image output from the generation unit 106, and stores the selected pre-deterioration image (output image) in memory 101b.

[0057] Based on the device usage information, when the electric scalpel and ultrasonic coagulation and cutting device are not in use, the selection unit 102 selects the input image with frame number n-1 stored in the memory 101a, and stores the selected pre-degraded image (input image) in the memory 101b.

[0058] FIG. 4 is a diagram showing the relationship between the duration of use of an electronic device and the density of smoke. The vertical axis of FIG. 4 corresponds to the density of smoke. The horizontal axis corresponds to time. In the example shown in FIG. 4, the time when the use of the electronic device starts is set as time t. The longer the duration of use of the electronic device, the thicker the smoke becomes.

[0059] Incidentally, the selection unit 102 may select an input image stored in the memory 101a while the smoke density is low, even if either the electric scalpel or the ultrasonic coagulating and cutting device is in use, and store the selected pre-deterioration image (input image) in the memory 101b. For example, the selection unit 102 receives information on each device in use in chronological order, and first selects an input image stored in the memory 101a from the time when either the electric scalpel or the ultrasonic coagulating and cutting device is in use until a predetermined time later, and stores the input image in the memory 101b.

[0060] ((Histogram conversion unit 103)) The histogram conversion unit 103 obtains the latest input image (input image with frame number n) from the memory 101a, and obtains the undegraded image from the memory 101b. In the explanation of the histogram conversion unit 103, the undegraded image is referred to as a "target image." The histogram conversion unit 103 converts the histogram h of the input image into S (s j ) is the histogram of the target image h T (t j ) to match the pixel values ​​of the input image.

[0061] Here, s j indicates the j-th pixel value in the input image. j indicates the j-th pixel value in the target image. For example, pixel values ​​of the input image and the target image range from 0 to 255. The input image after conversion is referred to as a “histogram-converted image.” The histogram conversion unit 103 outputs the histogram-converted image to the motion compensation unit 104.

[0062] Fig. 5 is a diagram showing an example of a histogram, in which the vertical axis corresponds to the frequency and the horizontal axis corresponds to the pixel value.

[0063] The process of the histogram conversion unit 103 will be described in detail below. The histogram conversion unit 103 normalizes the histogram by the number of pixels to obtain a probability density function. For example, the probability density function p S (s j ) is defined by equation (1). The histogram conversion unit 103 calculates the probability density function p S (s j ) is calculated.

[0064]

number

[0065] Probability density function of the target image p T (t j) is defined by equation (2). The histogram conversion unit 103 calculates the probability density function p T (t j ) is calculated.

[0066]

number

[0067] After calculating the probability density function, the histogram conversion unit 103 calculates the cumulative distribution function of the probability density function. For example, the cumulative distribution function F S (s k ) is defined by equation (3). The histogram conversion unit 103 calculates the cumulative distribution function F S (s k If the pixel values ​​of the input image range from 0 to 255, k=255.

[0068]

number

[0069] Cumulative distribution function of the target image F T (t k ) is defined by equation (4). The histogram conversion unit 103 calculates the cumulative distribution function F T (t k If the pixel values ​​of the target image range from 0 to 255, k=255.

[0070]

number

[0071] The cumulative distribution function of an image (input image or target image) represents a cumulative histogram of the image. FIG. 6 is a diagram showing an example of a cumulative histogram. The vertical axis of FIG. 6 corresponds to the cumulative frequency, and the horizontal axis corresponds to the pixel value. For example, when the pixel value s jThe cumulative frequency corresponding to pixel values ​​s0 to s j The cumulative frequency is calculated by dividing the number of

[0072] The histogram conversion unit 103 is T (s k ) inverse function F T -1 (s k ) and F S (s k )=F T (s k For example, the histogram conversion unit 103 converts each pixel value s of the input image into j (j=0 to 255) is calculated based on the equation (5) as the output pixel value o j The histogram-converted image H(I) is generated by converting the histograms into

[0073]

number

[0074] Here, the histogram conversion unit 103 does not necessarily need to perform the above-mentioned histogram conversion process on the entire image, but may perform the process on a specific region of the image or on a grid basis.

[0075] ((Motion Compensation Unit 104)) The motion compensation unit 104 is a processing unit that aligns the pre-degraded image based on the histogram-converted image and the pre-degraded image. For example, the histogram-converted image is an image converted from the input image of frame number n. The pre-degraded image is an image corresponding to the input image of frame number n-1 or an output image based on the input image of frame number n-1.

[0076] The motion compensation unit 104 compares the histogram converted image with the undegraded image to perform motion vector search (ME: Motion Estimation).The motion compensation unit 104 also performs motion compensation (MC: Motion Compensation) based on the result of the motion vector search to match the position of the subject in the undegraded image to the position of the subject in the histogram converted image.

[0077] The motion compensation unit 104 outputs the pre-degradation image after the alignment to the degradation estimation unit 105. Note that the motion compensation unit 104 may execute ME and MC using the technique disclosed in JP-A-2009-295029 and the like.

[0078] ((Deterioration estimation unit 105)) The degradation estimation unit 105 obtains the pre-degraded image that has been aligned from the motion compensation unit 104, and obtains the latest input image (input image with frame number n) from the memory 101a. In the explanation of the degradation estimation unit 105, the pre-degraded image that has been aligned is simply referred to as the pre-degraded image. The degradation estimation unit 105 is a processing unit that estimates degradation of the input image due to smoke (or mist) generated during surgery based on the pre-degraded image and the input image. The degradation estimation unit 105 outputs the estimation result to the generation unit 106.

[0079] The degradation estimation unit 105 includes a histogram conversion unit 105a, a correction map calculation unit 105b, and a correction map formation unit 105c. The histogram conversion unit 105a, the correction map calculation unit 105b, and the correction map formation unit 105c will be described below in this order.

[0080] ((Histogram conversion unit 105a)) The histogram conversion unit 105a obtains the latest input image (the input image with frame number n) from the memory 101a, and obtains the pre-degradation image from the motion compensation unit 104. The histogram conversion unit 105a converts the histogram h of the input image into S (s j ) is the histogram h of the target image (image before degradation) T (t jThe histogram conversion unit 105a generates a histogram-converted image by converting the pixel values ​​of the input image so that the pixel values ​​match the histogram-converted image H(I). The histogram conversion unit 105a outputs the histogram-converted image H(I) to the correction map calculation unit 105b.

[0081] The process in which the histogram conversion unit 105a generates a histogram-converted image based on an input image and a target image is similar to the process in the histogram conversion unit 103, and therefore a description thereof will be omitted.

[0082] ((Correction map calculation unit 105b)) The correction map calculation unit 105b calculates the correction amount map M based on the formula (6). As shown in the formula (6), the correction map calculation unit 105b calculates the difference between the histogram converted image H(I) and the input image I to calculate the contrast correction amount map M.

[0083] M = H(I) - I (6)

[0084] Incidentally, the correction map calculation unit 105b may calculate the correction amount map M by using a physical model of haze. For example, the correction map calculation unit 105b calculates the correction amount map M based on equation (7). In equation (7), the maximum pixel value among the pixel values ​​in the input image is set to A.

[0085] M = (I A) / (H(I)-A) (7)

[0086] It is assumed that whether the correction map calculation unit 105b calculates the correction amount map M using equation (6) or equation (7) is set in advance.

[0087] ((Correction map forming unit 105c)) The correction map forming unit 105c generates a shaped correction amount map F(I,M) by shaping the correction amount map with a guided filter using the input image as a guide image. The shaped correction amount map F(I,M) is the correction amount map M shaped to match the edges of the input image I. In this way, by using the shaped correction amount map F(I,M), it is possible to prevent image degradation around the edges that may occur when the positions of the correction amount map M and the input image I are misaligned.

[0088] The correction map forming unit 105c may generate the formed correction amount map F(I,M) by using a guided filter described in any one of Non-Patent Documents 1, 2, and 3 shown below.

[0089] Non-patent document 1: Kopf, Johannes, et al. "Joint bilateral upsampling." ACM Transactions on Graphics (ToG). Vol. 26. No. 3. ACM, 2007. Non-patent document 2: He, Kaiming, Jian Sun, and Xiaoou Tang. "Guided image filtering." European conference on computer vision. Springer, Berlin, Heidelberg, 2010. Non-Patent Document 3: Gastal, Eduardo SL, and Manuel M. Oliveira. "Domain transform for edge-aware image and video processing." ACM Transactions on Graphics (ToG). Vol. 30. No. 4. ACM, 2011.

[0090] The correction map forming unit 105c outputs the formed correction amount map F(I,M) as an estimated result of the deterioration to the generating unit 106. For example, in the formed correction amount map F(I,M), a pixel value of each pixel is defined.

[0091] ((Generation unit 106)) The generation unit 106 is a processing unit that generates an output image by correcting the input image based on the estimation result of the deterioration. The generation unit 106 outputs the output image to the display device 5155. The generation unit 106 also outputs the output image to the selection unit 102.

[0092] When the correction map M is calculated based on the formula (6), the generating unit 106 generates the output image O based on the formula (8). The formula (8) means that a process of adding the pixel value of a pixel in the input image I to the pixel value of a pixel in the same position in the shaped correction amount map F(I,M) is executed for each pixel.

[0093] O = F(I,M) + I (8)

[0094] On the other hand, when the correction map M is calculated based on the formula (7), the generation unit 106 generates the output image based on the formula (9).

[0095] O = (I A) / F (I, M) + A (9)

[0096] The above-mentioned selection unit 102, histogram conversion unit 103, motion compensation unit 104, degradation estimation unit 105, and generation unit 106 repeatedly execute the above-mentioned processes every time a new input image is stored in the memory 101a.

[0097] Fig. 7 is a diagram showing an example of an input image and an output image. In Fig. 7, smoke is generated in the input image 40, and light is reflected from the smoke-generated portion, and the transmittance is reduced, so that the image becomes whitish overall and the contrast of the background is reduced. The information processing device 100 executes the above-mentioned process on the input image 40 to generate an output image 41. In the output image 41, the contrast has returned to the original state, and the image is clear.

[0098] <<2.3. Operational Flow of Information Processing Device 100>> Next, a description will be given of the flow of operations of the information processing device 100 according to an embodiment of the present disclosure. Fig. 8 is a flowchart showing the flow of basic operations of the information processing device according to an embodiment of the present disclosure.

[0099] 8, the information processing device 100 receives an input image from the imaging device 10 and stores it in the memory 101a (step S101). The selection unit 102 of the information processing device 100 executes a selection process (step S102).

[0100] The histogram conversion unit 103 of the information processing device 100 generates a histogram-converted image based on the input image and the undegraded image (step S103). The motion compensation unit 104 of the information processing device 100 executes motion vector estimation (ME) and motion compensation (MC) on the undegraded image (step S104).

[0101] The degradation estimation unit 105 (histogram conversion unit 105a) of the information processing device 100 generates a histogram converted image based on the input image and the undegraded image after alignment (step S105). The degradation estimation unit 105 (correction map calculation unit 105b) calculates a correction amount map (step S106).

[0102] The deterioration estimation unit 105 (correction map forming unit 105c) generates a shaped correction amount map using the input image as a guide image (step S107). The generation unit 106 generates an output image based on the input image and the shaped correction amount map (step S108). The generation unit 106 outputs the output image to the display device 5155 (step S109).

[0103] If the information processing device 100 continues the process (step S110, Yes), the information processing device 100 proceeds to step S101. On the other hand, if the information processing device 100 does not continue the process (step S110, No), the information processing device 100 ends the process.

[0104] Next, a description will be given of the operation flow of the selection process shown in step S102 in Fig. 8. Fig. 9 is a flowchart showing the operation flow of the selection unit according to an embodiment of the present disclosure.

[0105] The selection unit 102 of the information processing device 100 receives the device-in-use information from the device-in-use monitoring apparatus 20 (step S201). The selection unit 102 determines whether the electronic device is in use (step S202).

[0106] If the electronic device is not in use (step S202, No), the selection unit 102 stores the input image in the memory 101b (step S203) and proceeds to step S205. On the other hand, if the electronic device is in use (step S202, Yes), the selection unit 102 stores the output image in the memory 101b (step S204).

[0107] If the information processing device 100 continues the process (step S205, Yes), the information processing device 100 proceeds to step S201. On the other hand, if the information processing device 100 does not continue the process (step S205, No), the information processing device 100 ends the selection process.

[0108] <<2.4. Effects of the information processing device according to the embodiment>> According to the information processing device 100 according to the first embodiment of the present disclosure, the degradation of the input image due to the generation of smoke is estimated based on the pre-degradation image and the input image, which is an image taken during surgery, and an output image is generated based on the estimation result and the input image. This makes it possible to correct the input image degraded by smoke in real time.

[0109] According to the information processing device 100, degradation is estimated by comparing the input image with the pre-degradation image, so that even if an object whose color resembles that of smoke is present in the endoscopic image, the object can be distinguished from the smoke, and an output image that is not affected by the color of the object can be generated.

[0110] According to the information processing device 100, a histogram-converted image is generated by converting pixel values ​​of the input image so that the histogram of the input image coincides with the histogram of the undegraded image, and a correction amount map is calculated based on the histogram-converted image and the input image. This makes it possible to accurately calculate the correction map even if the input image and the undegraded image are misaligned by about 10 or so pixels. This is because the histogram is a feature that does not have phase information and is robust to misalignment.

[0111] According to the information processing device 100, either the input image in the memory 101a or the output image of the generation unit 106 is selected based on the operating status of the electronic device, and stored in the memory 101b. In this way, by using the operating status of the device, it is possible to more appropriately select an undegraded image in which no smoke is generated.

[0112] In addition, the undegraded image stored in the memory 101b is the image immediately preceding the input image that is currently the subject of correction (the input image or the output image obtained by correcting the input image). Therefore, the time difference between the undegraded image to be compared and the input image can be reduced, and degradation of the image quality of the output image can be prevented.

[0113] According to the information processing device 100, a process of forming a correction amount map is executed using an input image as a guide image. This makes it possible to reduce image quality degradation around edges that may occur when a correction amount map is calculated in a situation where the input image and the undegraded image are misaligned by about several tens of pixels.

[0114] According to the information processing device 100, after converting the input image into a histogram-converted image, the histogram-converted image is compared with the undegraded image to perform motion compensation. This allows alignment to be performed with the contrast of the input image restored to a certain degree, and allows accurate alignment to be performed even when the degree of degradation of the contrast of the input image is large.

[0115] <<3. Hardware configuration>> The information processing device according to each embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 10, for example. The information processing device 100 according to the first embodiment will be described below as an example. Fig. 10 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0116] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0117] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0118] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0119] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0120] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0121] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded on the RAM 1200 to realize functions of the selection unit 102, the histogram conversion unit 103, the motion compensation unit 104, the degradation estimation unit 105, the generation unit 106, and the like. The HDD 1400 stores the generation program and the like according to the present disclosure. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0122] <4. Conclusion> The information processing device has a generation unit. The generation unit generates an output image by correcting the first image based on degradation of the first image due to a substance occurring during surgery, which is estimated based on a first image that is an image related to surgery and a second image that is an image from an earlier time than the first image. The generation unit corrects the first image based on degradation of the first image due to smoke, which is estimated based on the first image and the second image. This makes it possible to correct an input image that has been degraded by smoke in real time.

[0123] The generating unit corrects the first image based on a degradation estimated based on a past first image in which no smoke was generated, or the second image, which is a past output image generated by the generating unit, and the first image. This makes it possible to distinguish between the subject and the smoke even if an endoscopic image contains an object whose color resembles that of smoke, and to generate an output image that is not affected by the color of the object.

[0124] The information processing device further includes an estimation unit. The estimation unit estimates the deterioration of the first image due to a substance generated during the operation based on the first image and the second image. The generation unit generates an output image by correcting the first image based on the estimation result by the estimation unit. The estimation unit generates a converted image by converting pixel values ​​of the first image so that the histogram of the first image matches the histogram of the second image, and calculates a correction amount map based on the converted image and the first image. This makes it possible to accurately calculate the correction map even if the first image and the second image are misaligned by about a dozen pixels. This is because the histogram is a feature that does not have phase information and is robust to misalignment.

[0125] The generating unit generates the output image based on the first image and the correction amount map. The estimating unit calculates the difference between the converted image and the first image as the correction amount map. The estimating unit calculates the correction amount map based on the maximum pixel value among the pixel values ​​included in the first image, the converted image, and the first image. The estimating unit further executes a process of forming the correction amount map using the first image as a guide image. This makes it possible to reduce image quality degradation around edges that may occur when a correction amount map is calculated in a situation where the first image and the second image are misaligned by about several tens of pixels.

[0126] The image processing apparatus further includes a motion compensation unit that estimates a motion vector based on the first image and the second image and performs motion compensation of the second image based on the estimated motion vector to align the second image, and the estimation unit generates a converted image by converting pixel values ​​of the first image so that the histogram of the aligned second image matches. The image processing apparatus further includes a histogram conversion unit that generates a converted image by converting pixel values ​​of the first image so that the histogram of the first image matches the histogram of the second image, and the motion compensation unit estimates a motion vector based on the converted image and the second image and performs motion compensation of the second image based on the estimated motion vector to align the second image. This allows the alignment to be performed with the contrast of the first image restored to a certain extent, and allows accurate alignment to be performed even when the degree of deterioration of the contrast of the first image is large.

[0127] The information processing device further includes a selection unit that selects, as the second image, either the past output image generated by the generation unit or the past first image based on the operating status of an electronic device connected to the information processing device. In this way, by using the operating status of the device, it is possible to more appropriately select a pre-degradation image in which no smoke is generated.

[0128] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0129] The present technology can also be configured as follows. (1) a generating unit that generates an output image by correcting the first image based on deterioration of the first image due to a substance occurring during surgery, the deterioration being estimated based on a first image that is an image related to surgery and a second image that is an image taken earlier than the first image; An information processing device having the above configuration. (2) The information processing device described in (1) is characterized in that the generation unit corrects the first image based on degradation of the first image due to smoke estimated based on the first image and the second image. (3) The information processing device described in (2) is characterized in that the generation unit corrects the first image based on a deterioration estimated based on a past first image in which no smoke was generated, or the second image, which is a past output image generated by the generation unit, and the first image. (4) An estimation unit that estimates deterioration of the first image due to a substance generated during the operation based on the first image and the second image, The information processing device according to (1), (2) or (3), characterized in that the generation unit generates an output image by correcting the first image based on the estimation result by the estimation unit. (5) The information processing device described in (4) is characterized in that the estimation unit generates a converted image by converting pixel values ​​of the first image so that a histogram of the first image matches a histogram of the second image, and calculates a correction amount map based on the converted image and the first image. (6) The information processing device according to (5), wherein the generating unit generates the output image based on the first image and the correction amount map. (7) The information processing device described in (5) above, further comprising a motion compensation unit that aligns the second image by estimating a motion vector based on the first image and the second image and performing motion compensation of the second image based on the estimated motion vector, wherein the estimation unit generates a converted image by converting pixel values ​​of the first image so as to match a histogram of the aligned second image. (8) The information processing device described in (7) is characterized in that it further comprises a histogram conversion unit that generates a converted image by converting pixel values ​​of the first image so that the histogram of the first image matches the histogram of the second image, and the motion compensation unit estimates a motion vector based on the converted image and the second image, and performs motion compensation of the second image based on the estimated motion vector, thereby aligning the second image. (9) The information processing device according to (5), (6) or (7), wherein the estimation unit calculates a difference between the converted image and the first image as the correction amount map. (10) The information processing device described in (5), (6) or (7), characterized in that the estimation unit calculates the correction amount map based on a maximum pixel value among pixel values ​​included in the first image, the converted image, and the first image. (11) The information processing device according to any one of (5) to (9), wherein the estimation unit further executes a process of forming the correction amount map using the first image as a guide image. (12) The information processing device described in (1) is characterized in that it further has a selection unit that selects, as a second image, either a past output image generated by the generation unit or a past first image based on the operating status of an electronic device connected to the information processing device. (13) The computer An output image is generated by correcting a first image based on degradation of the first image due to a substance occurring during surgery, the degradation being estimated based on a first image related to surgery and a second image that is an image taken earlier than the first image. The generation method to perform the operation. (14) Computer, a generating unit that generates an output image by correcting the first image based on deterioration of the first image due to a substance occurring during surgery, the deterioration being estimated based on a first image that is an image related to surgery and a second image that is an image taken earlier than the first image; A generator to function as a. [Explanation of symbols]

[0130] 10. Imaging device 11 Imaging unit 12,22 Communications Department 20 Device monitoring equipment 21 Monitoring Department 30 Network 100 Information processing device 101a, 101b Memory 102 Selection section 103,105a Histogram conversion section 104 Motion compensation unit 105 Deterioration Estimation Unit 105b Correction map calculation unit 105c Correction map forming part 106 Generation part

Claims

1. A generating unit that generates an output image by correcting a first image based on deterioration of the first image due to smoke or mist, the deterioration being estimated based on a first image which is an endoscopic image or an in-vivo image and a second image which is an image taken earlier than the first image; and An information processing device having the above configuration.

2. The information processing device according to claim 1 , characterized in that the generation unit corrects the first image based on degradation of the first image due to smoke estimated based on the first image and the second image.

3. The information processing device according to claim 2, characterized in that the generation unit corrects the first image based on a deterioration estimated based on a past first image in which no smoke was generated, or the second image, which is a past output image generated by the generation unit, and the first image.

4. An estimation unit that estimates deterioration of the first image due to the smoke or the mist based on the first image and the second image, 4. The information processing device according to claim 1, wherein the generation unit generates an output image by correcting the first image based on a result of estimation by the estimation unit.

5. The information processing device according to claim 4, characterized in that the estimation unit generates a converted image by converting pixel values ​​of the first image so that a histogram of the first image matches a histogram of the second image, and calculates a correction amount map based on the converted image and the first image.

6. The information processing apparatus according to claim 5 , wherein the generating unit generates the output image based on the first image and the correction amount map.

7. The information processing device according to claim 5, further comprising a motion compensation unit that aligns the second image by estimating a motion vector based on the first image and the second image and performing motion compensation of the second image based on the estimated motion vector, wherein the estimation unit generates a converted image by converting pixel values ​​of the first image so as to match a histogram of the aligned second image.

8. The information processing device according to claim 7, further comprising a histogram conversion unit that generates a converted image by converting pixel values ​​of the first image so that the histogram of the first image matches the histogram of the second image, and the motion compensation unit aligns the second image by estimating a motion vector based on the converted image and the second image and performing motion compensation of the second image based on the estimated motion vector.

9. 8. The information processing apparatus according to claim 5, wherein the estimation unit calculates a difference between the converted image and the first image as the correction amount map.

10. The information processing device according to any one of claims 5 to 7, characterized in that the estimation unit calculates the correction amount map based on a maximum pixel value among pixel values ​​included in the first image, the converted image, and the first image.

11. 10. The information processing apparatus according to claim 5, wherein the estimation unit further executes a process of forming the correction amount map using the first image as a guide image.

12. The information processing device according to claim 1, further comprising a selection unit that selects, as the second image, either a past output image generated by the generation unit or a past first image based on the operating status of an electronic device connected to the information processing device.

13. The computer An output image is generated by correcting a first image based on deterioration of the first image caused by smoke or mist, the deterioration being estimated based on a first image which is an endoscopic image or an in-vivo image and a second image which is an image older than the first image. The generation method to perform the operation.

14. Computer, a generating unit that generates an output image by correcting a first image based on deterioration of the first image caused by smoke or mist, the deterioration being estimated based on a first image which is an endoscopic image or an in-vivo image and a second image which is an image older than the first image; A generator to function as a.

Images