Image processing device, treatment system, and image processing method

Abstract

Provided are an image processing device, a treatment system, and an image processing method which can continue treatment for a treatment unit even when the visual field of an endoscope has deteriorated. The image processing device comprises: a first image acquisition unit which acquires first image data including an area in which a living body is treated by an energy treatment tool; a first detection unit which detects a change in gray scale from at least a portion of an area of a first image corresponding to the first image data; a first corrected image generation unit which generates, on the basis of the detection result of the first detection unit, first corrected image data by correcting the gray scale of the first image; and a display image generation unit which generates a display image based on the first corrected image data.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, a treatment system, and an image processing method. [Background technology]

[0002] In arthroscopic surgery, a technique is known in which the inside of the joint is inflated with an irrigating fluid such as saline using an irrigation device to ensure a field of view and then the treatment area is treated (see, for example, Patent Document 1). In this technique, the hammering action of an ultrasonic treatment instrument breaks down the bone, generating bone powder (bone shavings) and cerebrospinal fluid, so the irrigating fluid is used to send the bone powder and cerebrospinal fluid out of the field of view of the endoscope to ensure a field of view of the treatment area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4564595 Summary of the Invention [Problem to be solved by the invention]

[0004] In arthroscopic surgery, when bones are continuously crushed by the hammering action of an ultrasonic treatment instrument, a large amount of bone powder is generated. This bone powder disperses in the irrigation fluid, making the fluid cloudy and obstructing the field of view of the arthroscope used to observe the treatment area, making it difficult to see the treatment area.

[0005] However, in the above-mentioned Patent Document 1, if the field of view of the endoscope observing the treatment area becomes cloudy and deteriorates, the perfusion fluid will expel bone powder from the field of view of the endoscope, and treatment on the treatment area must be stopped and the surgeon must wait until the field of view of the endoscope improves, which extends the treatment time and places a burden on both the surgeon and the patient.

[0006] The present disclosure has been made in consideration of the above, and aims to provide an image processing device, a treatment system, and an image processing method that enable treatment of a treatment area to be continued even if the field of view of the endoscope deteriorates. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the image processing device according to the present disclosure includes a first image acquisition unit that acquires first image data including an area where a living body is to be treated by an energy treatment tool, a first detection unit that detects a change in gradation from at least a portion of an area of the first image that corresponds to the first image data, a first corrected image generation unit that generates first corrected image data by correcting the gradation of the first image based on the detection result of the first detection unit, and a display image generation unit that generates a display image based on the first corrected image data.

[0008] In addition, the image processing device according to the present disclosure includes a first image acquisition unit that acquires first image data including an area where a living body is to be treated by an energy treatment tool, a second image acquisition unit that acquires second image data having a different wavelength from the first image, a detection unit that detects a change in gradation from at least a portion of the area of the first image, a correction image generation unit that generates corrected image data by performing gradation correction on the second image based on the detection result of the detection unit, and a display image generation unit that generates a display image based on the correction image data.

[0009] In addition, a treatment system according to the present disclosure includes an energy treatment device that can be inserted into a subject and can treat a treatment target area, an endoscope that can be inserted into the subject and can capture an image of at least the treatment target area to generate first image data, and an image processing device that performs image processing on the first image data and outputs the image to a display device, wherein the image processing device includes a first image acquisition unit that acquires the first image data, a first detection unit that detects a change in gradation from at least a partial area of a first image corresponding to the first image data, a first corrected image generation unit that generates first corrected image data by correcting the gradation of the first image based on a detection result of the first detection unit, and a display image generation unit that generates a display image based on the first corrected image data.

[0010] Furthermore, the image processing method according to the present disclosure is an image processing method executed by an image processing device provided with a processor having hardware, in which the processor acquires first image data including an area of a living body to be treated by an energy treatment device, detects a change in gradation from at least a portion of an area of the first image corresponding to the first image data, generates first corrected image data by gradation correcting the first image based on the detection result of the change in gradation from at least a portion of the area of the first image, and generates a display image based on the first corrected image data. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to achieve the effect of continuing treatment on the treatment site even when the field of view of the endoscope deteriorates. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a treatment system according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a state in which a bone hole is formed by the ultrasonic probe according to the first embodiment of the present disclosure. [Figure 3A]FIG. 3A is a schematic diagram illustrating a schematic configuration of an ultrasound probe according to the first embodiment of the present disclosure. [Figure 3B] FIG. 3B is a schematic view of the direction of arrow A in FIG. 3A. [Figure 4] FIG. 4 is a block diagram illustrating an outline of the functional configuration of the entire treatment system according to the first embodiment of the present disclosure. [Figure 5] FIG. 5 is a block diagram illustrating a detailed functional configuration of the endoscope apparatus according to the first embodiment of the present disclosure. [Figure 6A] FIG. 6A is a diagram showing a state in which the field of view of the endoscope according to the first embodiment of the present disclosure is good. [Figure 6B] FIG. 6B is a diagram illustrating a state in which the field of view of the endoscope according to the first embodiment of the present disclosure is poor. [Figure 7] FIG. 7 is a block diagram illustrating a detailed functional configuration of the processing device according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram illustrating a detailed functional configuration of the perfusion device according to the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a block diagram illustrating a detailed functional configuration of the illumination device according to the first embodiment of the present disclosure. [Figure 10] FIG. 10 is a block diagram illustrating a functional configuration of the imaging device according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram schematically illustrating a configuration of a pixel unit according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram schematically illustrating the configuration of a color filter according to the first embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram schematically illustrating the sensitivity and wavelength band of each filter according to the first embodiment of the present disclosure. [Figure 14] FIG. 14 is a block diagram illustrating a detailed functional configuration of the image processing unit according to the first embodiment of the present disclosure. [Figure 15] FIG. 15 is a block diagram illustrating a detailed functional configuration of the first corrected image generating unit according to the first embodiment of the present disclosure. [Figure 16]FIG. 16 is a flowchart illustrating an outline of a treatment performed by an operator using the treatment system according to the first embodiment of the present disclosure. [Figure 17] FIG. 17 is a flowchart illustrating an overview of processing executed by the endoscope control device according to the first embodiment of the present disclosure in a cutting procedure. [Figure 18] FIG. 18 is a flowchart showing a detailed outline of the turbidity response control process of FIG. [Figure 19] Figure 19 is a diagram showing an example of a first image in the field of view of an endoscope that is continuous in time and that is generated by a display image generation unit based on a first image and output to a display device when the turbidity correction processing by the first corrected image generation unit according to embodiment 1 of the present disclosure has not been performed. [Figure 20] FIG. 20 is a diagram showing an example of temporally consecutive first corrected images in the field of view of an endoscope that are generated by a display image generating unit based on the first corrected images and output to a display device when turbidity correction processing is performed by the first corrected image generating unit according to embodiment 1 of the present disclosure. [Figure 21] Figure 21 is a diagram showing an example of a temporally consecutive second corrected image in the field of view of an endoscope that is generated by a display image generation unit based on the second corrected image and output to a display device when edge enhancement processing is performed by the second corrected image generation unit according to embodiment 1 of the present disclosure. [Figure 22] Figure 22 is a diagram showing an example of a composite image in the field of view of an endoscope that is continuous in time and that is generated by a display image generation unit based on the composite image and output to a display device when a synthesis process is performed by the composite image generation unit according to embodiment 1 of the present disclosure. [Figure 23] FIG. 23 is a diagram illustrating an example of temporally consecutive images in the field of view of an endoscope, in which the display image generating unit according to the first embodiment of the present disclosure outputs the first corrected image and the second corrected image to the display device. [Figure 24] FIG. 24 is a block diagram illustrating a functional configuration of an endoscope according to the second embodiment of the present disclosure. [Figure 25] FIG. 25 is a diagram illustrating an example of a composite image generated by a composite image generating unit according to the second embodiment of the present disclosure. [Figure 26] FIG. 26 is a block diagram illustrating a functional configuration of an endoscope according to the third embodiment of the present disclosure. [Figure 27] FIG. 27 is a block diagram illustrating a functional configuration of an illumination device according to the third embodiment of the present disclosure. [Figure 28] FIG. 28 is a schematic diagram illustrating a schematic configuration of an illumination unit according to the third embodiment of the present disclosure. [Figure 29] FIG. 29 is a diagram illustrating the relationship between the transmission characteristics and wavelength bands of the red, green, and blue filters according to the third embodiment of the present disclosure. [Figure 30] FIG. 30 is a diagram illustrating the relationship between the transmission characteristics and wavelength bands of the IR filter according to the third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. Furthermore, each drawing referred to in the following description merely shows a schematic representation of the shape, size, and positional relationship to the extent that the contents of the present disclosure can be understood. In other words, the present disclosure is not limited to only the shape, size, and positional relationship exemplified in each drawing. Furthermore, in the following description, the same parts in the drawings will be denoted by the same reference numerals.

[0014] (Embodiment 1) [General configuration of processing system] FIG. 1 is a diagram showing a schematic configuration of a treatment system 1 according to the first embodiment. The treatment system 1 shown in FIG. 1 treats biological tissue such as bone by applying ultrasonic vibrations to the biological tissue. Here, treatment refers to, for example, removal or cutting of biological tissue such as bone. Note that FIG. 1 illustrates a treatment system for performing anterior cruciate ligament reconstruction as the treatment system 1.

[0015] The treatment system 1 shown in FIG. 1 includes an endoscope apparatus 2, a treatment apparatus 3, a guiding device 4, a perfusion apparatus 5, and an illumination apparatus 6.

[0016] [Configuration of the endoscope device] First, the configuration of the endoscope device 2 will be described. The endoscope device 2 includes an endoscope 201 , an endoscope control device 202 , and a display device 203 .

[0017] The endoscope 201 has a distal end portion of an insertion section 211 inserted into the joint cavity C1 through a first portal P1 that communicates between the inside of the joint cavity C1 of the knee joint J1 of the subject and the outside of the skin. The endoscope 201 illuminates the inside of the joint cavity C1, takes in illumination light (subject image) reflected within the joint cavity C1, and captures the subject image to generate image data.

[0018] The endoscope control device 202 performs various image processing on image data captured by the endoscope 201, and displays a display image corresponding to the image data after this image processing on the display device 203. The endoscope control device 202 is connected to the endoscope 201 and the display device 203 by wire or wirelessly.

[0019] The display device 203 receives data, image data (display image), audio data, etc. transmitted from each device constituting the treatment system 1 via the endoscope control device 202, and displays, notifies, and outputs a display image according to the received data. The display device 203 is configured using a display panel made of liquid crystal or organic EL (Electro-Luminescence).

[0020] [Configuration of processing device] Next, the configuration of the processing device 3 will be described. The treatment device 3 includes a treatment tool 301 , a treatment tool control device 302 , and a foot switch 303 .

[0021] The treatment tool 301 includes a treatment tool body 311, an ultrasonic probe 312 (see FIG. 2, which will be described later), and a sheath 313.

[0022] The treatment tool body 311 is formed in a cylindrical shape. The treatment tool body 311 also houses an ultrasonic transducer 312a (see FIG. 2, which will be described later) that is made up of a bolt-clamped Langevin-type transducer and generates ultrasonic vibrations in response to supplied driving power.

[0023] The treatment tool control device 302 supplies driving power to the ultrasonic transducer 312a in response to an operator's operation of the foot switch 303. The supply of driving power is not limited to the operation of the foot switch 303, and may be performed in response to, for example, an operation of an operation unit (not shown) provided on the treatment tool 301.

[0024] The foot switch 303 is an input interface that is operated by the operator with his / her foot when driving the ultrasonic probe 312 .

[0025] Next, the ultrasonic probe 312 will be described. Fig. 2 is a diagram showing how a bone hole 101 is formed by an ultrasonic probe 312. Fig. 3A is a schematic diagram showing a general configuration of the ultrasonic probe 312. Fig. 3B is a schematic diagram seen in the direction of arrow A in Fig. 3A.

[0026] As shown in FIGS. 2, 3A, and 3B, the ultrasonic probe 312 is made of, for example, a titanium alloy and has a substantially cylindrical shape. The proximal end of the ultrasonic probe 312 is connected to an ultrasonic vibrator 312a inside the treatment instrument main body 311. The ultrasonic probe 312 transmits ultrasonic vibrations generated by the ultrasonic vibrator 312a from the proximal end to the distal end. Specifically, the ultrasonic vibrations in the first embodiment are longitudinal vibrations along the longitudinal direction of the ultrasonic probe 312 (the vertical direction in FIG. 2). The distal end of the ultrasonic probe 312 is provided with the ultrasonic vibrator 312a, as shown in FIG. 2.

[0027] The sheath 313 is formed in a cylindrical shape that is longer and thinner than the treatment instrument body 311, and covers a part of the outer periphery of the ultrasonic probe 312 from the treatment instrument body 311 to an arbitrary length.

[0028] The ultrasonic vibrator 312a of the ultrasonic probe 312 in the treatment instrument 301 configured in this manner is inserted into the joint cavity C1 while being guided by the guiding device 4 inserted into the joint cavity C1 through a second portal P2 that connects the inside of the joint cavity C1 with the outside of the skin.

[0029] Next, the treatment tool 301 generates ultrasonic vibrations while the ultrasonic vibrator 312a of the ultrasonic probe 312 is in contact with the bone treatment target area 100, and the hammering action causes the part of the bone that mechanically collides with the ultrasonic vibrator 312a to be crushed into fine particles (see Figure 2).

[0030] Thereafter, when the operator presses the ultrasonic vibrator 312a of the ultrasonic probe 312 into the treatment target portion 100, the ultrasonic vibrator 312a advances into the treatment target portion 100 while crushing the bone. As a result, a bone hole 101 is formed in the treatment target portion 100.

[0031] Further, at the proximal end of the treatment tool body 311, a circuit board 317 on which a posture detection section 314, a CPU (Central Processing Unit) 315, and a memory 316 are mounted is provided (see FIGS. 3A and 3B).

[0032] The attitude detection unit 314 includes a sensor that detects the rotation and movement of the treatment tool 301. The attitude detection unit 314 detects movement in three mutually orthogonal axial directions, including an axis parallel to the longitudinal axis of the ultrasound probe 312, and rotation around each axis. The treatment tool control device 302 determines that the treatment tool 301 is stationary if the detection result of the attitude detection unit 314 does not change for a certain period of time. The attitude detection unit 314 is composed of, for example, a three-axis angular velocity sensor (gyro sensor) and an acceleration sensor.

[0033] The CPU 315 controls the operation of the attitude detection unit 314 and transmits and receives information to and from the treatment tool control device 302. The CPU 315 reads a program stored in the memory 316 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby enabling the hardware and software to work together and realizing a functional module that meets a predetermined purpose.

[0034] [Configuration of the guiding device] Next, the configuration of the guiding device 4 will be described. In FIG. 1, the guiding device 4 is inserted into the joint cavity C1 through the second portal P2, and guides the insertion of the tip portion of the ultrasonic probe 312 of the treatment tool 301 into the joint cavity C1.

[0035] The guiding device 4 includes a guide body 401, a handle portion 402, and a drainage portion 403 with a cock.

[0036] The guide body 401 has a cylindrical shape and has a through-hole 401a therein through which the ultrasonic probe 312 is inserted (see FIG. 1). The guide body 401 restricts the advancement of the ultrasonic probe 312 inserted into the through-hole 401a in a fixed direction and guides the movement of the ultrasonic probe 312. In the first embodiment, the cross-sectional shapes of the outer and inner peripheral surfaces of the guide body 401 perpendicular to the central axis are each approximately circular. Furthermore, the guide body 401 becomes thinner toward the tip. In other words, the tip surface 401b of the guide body 401 is an inclined surface that intersects obliquely with the central axis.

[0037] The cock-equipped drainage section 403 is provided on the outer peripheral surface of the guide main body 401 and has a cylindrical shape that communicates with the inside of the guide main body 401. One end of a drainage tube 505 of the perfusion device 5 is connected to the cock-equipped drainage section 403, forming a flow path that communicates between the guide main body 401 and the drainage tube 505 of the perfusion device 5. This flow path is configured to be openable and closable by operating a cock (not shown) provided in the cock-equipped drainage section 403.

[0038] [Configuration of the perfusion device] Next, the configuration of the perfusion device 5 will be described. In FIG. 1, the perfusion device 5 delivers a perfusion fluid such as sterilized physiological saline into the joint cavity C1 and also discharges the perfusion fluid to the outside of the joint cavity C1.

[0039] The perfusion device 5 includes a liquid source 501, a liquid supply tube 502, a liquid supply pump 503, a drain bottle 504, a drain tube 505, and a drain pump 506 (see FIG. 1).

[0040] Fluid source 501 contains an irrigation fluid therein. Fluid source 501 is connected to fluid supply tube 502. The irrigation fluid is sterilized physiological saline or the like. Fluid source 501 is configured using, for example, a bottle or the like.

[0041] The liquid supply tube 502 has one end connected to the liquid source 501 and the other end connected to the endoscope 201 .

[0042] The fluid supply pump 503 supplies the perfusion fluid from the fluid source 501 to the endoscope 201 through the fluid supply tube 502. The perfusion fluid supplied to the endoscope 201 is supplied into the joint cavity C1 from a fluid supply hole formed at the tip of the insertion section 211.

[0043] The drainage bottle 504 stores the irrigation fluid discharged to the outside of the joint cavity C1. A drainage tube 505 is connected to the drainage bottle 504.

[0044] The drainage tube 505 has one end connected to the guiding device 4 and the other end connected to the drainage bottle 504 .

[0045] The drainage pump 506 drains the perfusion fluid in the joint cavity C1 from the guiding device 4 inserted into the joint cavity C1 through the flow path of the drainage tube 505 into the drainage bottle 504. Although the first embodiment uses the drainage pump 506 for explanation, this is not limiting and a suction device provided in the facility may also be used.

[0046] [Configuration of lighting device] Next, the configuration of the lighting device 6 will be described. 1, the illumination device 6 has two light sources that emit two illumination lights having different wavelength bands. The two illumination lights are, for example, white light, which is visible light, and infrared light, which is invisible light. The illumination lights from the illumination device 6 are propagated to the endoscope 201 via a light guide and emitted from the tip of the endoscope 201.

[0047] [Functional configuration of the entire treatment system] Next, the functional configuration of the entire treatment system will be described. FIG. 4 is a block diagram showing an outline of the functional configuration of the entire treatment system 1. As shown in FIG. In addition to the configuration described above (see FIG. 1), the treatment system 1 shown in FIG. 4 further includes a network control device 7 that controls communication throughout the system, and a network server 8 that stores various data.

[0048] The network control device 7 is communicably connected to the endoscope device 2, treatment device 3, perfusion device 5, illumination device 6, and network server 8. While Fig. 4 illustrates a case where the devices are connected wirelessly, they may also be connected by wire. The detailed functional configurations of the endoscope device 2, treatment device 3, perfusion device 5, and illumination device 6 will be described below.

[0049] The network server 8 is communicably connected to the endoscope device 2, the treatment device 3, the perfusion device 5, the illumination device 6, and the network control device 7. The network server 8 stores various data of each device constituting the treatment system 1. The network server 8 is configured using a processor having hardware such as a CPU, and memories such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0050] [Functional configuration of the endoscope device] Next, the functional configuration of the above-mentioned endoscope device 2 will be described. FIG. 5 is a block diagram showing a detailed functional configuration of the endoscope device 2. As shown in FIG. As shown in FIGS. 4 and 5, the endoscope device 2 includes an endoscope control device 202, a display device 203, an imaging unit 204 provided in the endoscope 201, and an operation input unit 205.

[0051] The endoscope control device 202 includes an imaging processing unit 221, an image processing unit 222, a turbidity detection unit 223, an input unit 226, a CPU 227, a memory 228, a wireless communication unit 229, a distance sensor driving circuit 230, a distance data memory 231, and a communication interface 232.

[0052] The imaging processing unit 221 has an imaging element drive control circuit 221a that controls the drive of the imaging element 2241 included in the imaging unit 204 provided in the endoscope 201, and an imaging element signal control circuit 221b that is provided in a patient circuit 202b electrically insulated from the primary circuit 202a and controls the signal of the imaging element 224a. The imaging element drive control circuit 221a is provided in the primary circuit 202a. The imaging element signal control circuit 221b is provided in a patient circuit 202b electrically insulated from the primary circuit 202a.

[0053] The image processing unit 222 performs predetermined image processing on image data (RAW data) input via a bus and outputs the result to the display device 203. The image processing unit 222 is configured using a processor having hardware such as a DSP (Digital Signal Processor) or FPGA (Field-Programmable Gate Array). The image processing unit 222 reads a program stored in a memory 228 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby enabling the hardware and software to work together and realize a functional module that meets a predetermined purpose. The detailed functional configuration of the image processing unit 222 will be described later.

[0054] The turbidity detection unit 223 detects turbidity of the field of view of the endoscope 201 in the joint cavity C1 based on information about turbidity of the field of view of the endoscope 201. Here, the information about turbidity is, for example, a value obtained from image data generated by the endoscope 201, a physical property value (turbidity) of the perfusion fluid, impedance acquired from the treatment device 3, etc.

[0055] FIG. 6A is a diagram showing a state in which the field of view of the endoscope 201 is good. FIG. 6B is a diagram showing a state in which the field of view of the endoscope 201 is poor. Note that each of FIGS. 6A and 6B is a diagram showing a displayed image corresponding to image data showing the field of view of the endoscope 201 when the surgeon forms a bone hole in the femoral lateral condyle 900. Of these, FIG. 6B is a diagram showing a state in which the field of view of the endoscope 201 is blurred due to bone being crushed into fine particles by driving the ultrasonic probe 312. Note that in FIG. 6B, fine bones are represented by dots.

[0056] In FIG. 5, the input unit 226 receives the input of a signal input by the operation input unit 205 and the input of a signal from each device constituting the treatment system 1.

[0057] The CPU 227 comprehensively controls the operation of the endoscope control device 202. The CPU 227 reads out a program stored in the memory 228 into a working area of the memory and executes it, and controls each component part through the execution of the program by the processor, thereby controlling the operation of each part of the endoscope control device 202 through cooperation between hardware and software.

[0058] The memory 228 stores various information necessary for the operation of the endoscope control device 202, various programs executed by the endoscope control device 202, image data captured by the imaging unit 204, etc. The memory 228 is configured using, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a frame memory, etc.

[0059] The wireless communication unit 229 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0060] The distance sensor driving circuit 230 drives a distance sensor (not shown) that measures the distance to a predetermined object in an image captured by the imaging unit 204. In the first embodiment, the distance sensor may be provided in the imaging element 2241. In this case, the imaging element 2241 may be provided with phase difference pixels capable of measuring the distance from the imaging element 2241 to the predetermined object, instead of effective pixels. Of course, a ToF (Time of Flight) sensor or the like may be provided near the tip of the endoscope 201.

[0061] Distance data detected by the distance sensor is stored in the distance data memory 231. The distance data memory 231 is configured using, for example, RAM and ROM.

[0062] The communication interface 232 is an interface for communicating with the image capturing unit 204 .

[0063] Of the above-described components, all except the imaging element signal control circuit 221b are provided in the primary circuit 202a and are interconnected by bus wiring.

[0064] The imaging unit 204 is provided in the endoscope 201. The imaging unit 204 has an imaging element 2241, a CPU 242, and a memory 243.

[0065] Under the control of the CPU 242, the image sensor 2241 generates image data by capturing an image of a subject formed by one or more optical systems (not shown), and outputs the generated image data to the endoscope control device 202. The image sensor 2241 is configured using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0066] The CPU 242 comprehensively controls the operation of the imaging unit 204. The CPU 242 reads out a program stored in the memory 243 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby controlling the operation of the imaging unit 204 through cooperation between hardware and software.

[0067] The memory 243 stores various information necessary for the operation of the imaging unit 204, various programs executed by the endoscope 201, image data generated by the imaging unit 204, etc. The memory 243 is configured using RAM, ROM, frame memory, etc.

[0068] The operation input unit 205 is configured using input interfaces such as a mouse, keyboard, touch panel, and microphone, and receives operation inputs of the endoscope device 2 from the operator.

[0069] [Functional configuration of the processing device] Next, the functional configuration of the processing device 3 will be described. FIG. 7 is a block diagram showing a detailed functional configuration of the processing device 3. As shown in FIG. As shown in FIGS. 4 and 7, the treatment device 3 includes a treatment tool 301, a treatment tool control device 302, and an input / output unit 304.

[0070] The treatment tool 301 includes an ultrasonic transducer 312 a , a posture detection unit 314 , a CPU 315 , and a memory 316 .

[0071] The attitude detection unit 314 detects the attitude of the treatment tool 301 and outputs the detection result to the CPU 315. The attitude detection unit 314 is configured using at least one of an acceleration sensor and an angular velocity sensor.

[0072] The CPU 315 comprehensively controls the operation of the treatment tool 301 including the ultrasonic transducer 312a. The CPU 315 reads out a program stored in the memory 316 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby allowing the hardware and software to work together and realizing a functional module that meets a predetermined purpose.

[0073] The memory 316 stores various information necessary for the operation of the treatment tool 301, various programs executed by the treatment tool 301, and identification information for identifying the type, manufacturing date, performance, etc. of the treatment tool 301.

[0074] The treatment tool control device 302 includes a primary circuit 321, a patient circuit 322, a transformer 323, a first power supply 324, a second power supply 325, a CPU 326, a memory 327, a wireless communication unit 328, a communication interface 329, and an impedance detection unit 330.

[0075] The primary circuit 321 generates power to be supplied to the treatment tool 301. The patient circuit 322 is electrically insulated from the primary circuit 321. The transformer 323 electromagnetically connects the primary circuit 321 and the patient circuit 322. The first power supply 324 is a high-voltage power supply that supplies drive power to the treatment tool 301. The second power supply 325 is a low-voltage power supply that supplies driving power to the control circuit in the treatment tool control device 302 .

[0076] The CPU 326 comprehensively controls the operation of the treatment tool control device 302. The CPU 326 reads out a program stored in the memory 327 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby controlling the operation of each part of the treatment tool control device 302 through cooperation between hardware and software.

[0077] The memory 327 stores various information necessary for the operation of the treatment tool control device 302, various programs executed by the treatment tool control device 302, etc. The memory 327 is configured using RAM, ROM, etc.

[0078] The wireless communication unit 328 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi (registered trademark) and Bluetooth (registered trademark).

[0079] The communication interface 329 is an interface for communicating with the treatment tool 301 .

[0080] The impedance detection unit 330 detects the impedance when the treatment tool 301 is driven, and outputs the detection result to the CPU 326. Specifically, the impedance detection unit 330 is electrically connected, for example, between the first power source 324 and the primary circuit 321, and detects the impedance of the treatment tool 301 based on the frequency of the first power source 324 and outputs the detection result to the CPU 326.

[0081] The input / output unit 304 is configured using input interfaces such as a mouse, keyboard, touch panel, and microphone, and output interfaces such as a monitor and speaker, and outputs operation inputs of the endoscopic device 2 by the surgeon and various information to be notified to the surgeon.

[0082] [Functional configuration of the perfusion device] Next, the functional configuration of the perfusion device 5 will be described. FIG. 8 is a block diagram showing the detailed functional configuration of the perfusion device 5. As shown in Figures 4 and 8, the perfusion device 5 includes a fluid supply pump 503, a drainage pump 506, a fluid supply control unit 507, a drainage control unit 508, an input unit 509, a CPU 510, a memory 511, a wireless communication unit 512, a communication interface 513, a pump CPU 514, a pump memory 515, and a turbidity detection unit 516.

[0083] The liquid supply control unit 507 includes a first drive control unit 571 , a first drive power generation unit 572 , a first transformer 573 , and a liquid supply pump drive circuit 574 .

[0084] The first drive control unit 571 controls the driving of the first drive power generation unit 572 and the liquid feed pump drive circuit 574 .

[0085] The first drive power generation unit 572 generates drive power for the liquid feed pump 503 and supplies this drive power to the first transformer 573 .

[0086] The first transformer 573 electromagnetically connects the first drive power generation unit 572 and the liquid feed pump drive circuit 574 together.

[0087] In fluid supply control unit 507 configured in this manner, first drive control unit 571, first drive power generation unit 572, and first transformer 573 are provided in primary circuit 5a. In addition, fluid supply pump drive circuit 574 is provided in patient circuit 5b that is electrically insulated from primary circuit 5a.

[0088] The drainage control unit 508 includes a second drive control unit 581, a second drive power generation unit 582, a second transformer 583, and a drainage pump drive circuit 584.

[0089] The second drive control unit 581 controls the driving of the second drive power generation unit 582 and the drainage pump drive circuit 584 .

[0090] The second driving power generating unit 582 generates driving power for the drainage pump 506 and supplies the generated driving power to the second transformer 583 .

[0091] The second transformer 583 electromagnetically connects the second drive power generating unit 582 and the drainage pump drive circuit 584 together.

[0092] In drainage control unit 508 configured in this manner, second drive control unit 581, second drive power generation unit 582, and second transformer 583 are provided in primary circuit 5a. Furthermore, drainage pump drive circuit 584 is provided in patient circuit 5b that is electrically insulated from primary circuit 5a.

[0093] The input unit 509 receives operation inputs (not shown) and signal inputs from the devices that make up the treatment system 1, and outputs the received signals to the CPU 510 and the pump CPU 514.

[0094] The CPU 510 and the pump CPU 514 work together to comprehensively control the operation of the perfusion device 5. The CPU 510 reads a program stored in the memory 511 into a working area of the memory and executes it, and controls each component part through the execution of the program by the processor, thereby controlling the operation of each part of the perfusion device 5 through cooperation between hardware and software.

[0095] The memory 511 stores various information necessary for the operation of the perfusion device 5 and various programs executed by the perfusion device 5. The memory 511 is configured using RAM, ROM, and the like.

[0096] The wireless communication unit 512 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi or Bluetooth.

[0097] The communication interface 513 is an interface for communicating with the liquid feed pump 503 and the endoscope 201 .

[0098] The pump memory 515 stores various information necessary for the operation of the liquid feed pump 503 and the liquid discharge pump 506 and various programs executed by the liquid feed pump 503 and the liquid discharge pump 506 .

[0099] The turbidity detection unit 516 detects the turbidity of the perfusion fluid based on one or more of the physical properties, absorbance, impedance, and resistance of the perfusion fluid flowing in the drainage tube 505, and outputs the detection result to the CPU 510.

[0100] In the perfusion device 5 configured as above, an input unit 509, a CPU 510, a memory 511, a wireless communication unit 512, a communication interface 513, and a turbidity detection unit 516 are provided in the primary circuit 5a. Furthermore, an in-pump CPU 514 and an in-pump memory 515 are provided in the pump 5c. The in-pump CPU 514 and the in-pump memory 515 may be provided in the vicinity of the liquid feed pump 503 or in the vicinity of the drainage pump 506.

[0101] [Functional configuration of lighting device] Next, the functional configuration of the lighting device 6 will be described. FIG. 9 is a block diagram showing a detailed functional configuration of the lighting device 6. As shown in FIG. As shown in Figures 4 and 9, the lighting device 6 includes a first lighting control unit 601, a second lighting control unit 602, a first lighting device 603, a second lighting device 604, an input unit 605, a CPU 606, a memory 607, a wireless communication unit 608, a communication interface 609, a CPU 610 in the lighting circuit, and a memory 630 in the lighting circuit.

[0102] The first illumination control unit 601 has a first drive control unit 611 , a first drive power generation unit 612 , a first controller 613 , and a first drive circuit 614 .

[0103] The first drive control unit 611 controls the driving of the first drive power generation unit 612 , the first controller 613 and the first drive circuit 614 .

[0104] The first drive power generation section 612 generates drive power for the first lighting device 603 under the control of the first drive control section 611 , and outputs this drive power to the first controller 613 .

[0105] The first controller 613 controls the first drive circuit 614 in accordance with the drive power input from the first drive power generation unit 612 , thereby controlling the light output of the first illumination device 603 .

[0106] The first driving circuit 614 drives the first lighting device 603 under the control of the first controller 613 to output illumination light.

[0107] In the first illumination control section 601 configured in this manner, a first drive control section 611, a first drive power generation section 612, and a first controller 613 are provided in a primary circuit 6a. Also, a first drive circuit 614 is provided in a patient circuit 6b that is electrically insulated from the primary circuit 6a.

[0108] The second illumination control unit 602 includes a second drive control unit 621 , a second drive power generation unit 622 , a second controller 623 , and a second drive circuit 624 .

[0109] The second drive control unit 621 controls the driving of the second drive power generation unit 622 , the second controller 623 and the second drive circuit 624 .

[0110] The second drive power generation section 622 generates drive power for the second lighting device 604 under the control of the second drive control section 621 , and outputs this drive power to the second controller 623 .

[0111] The second controller 623 controls the second drive circuit 624 in accordance with the drive power input from the second drive power generation section 622, thereby controlling the light output of the second illumination device 604.

[0112] The second driving circuit 624 drives the second lighting device 604 under the control of the second controller 623 to output illumination light.

[0113] In second illumination control section 602 configured in this manner, second drive control section 621, second drive power generation section 622, and second controller 623 are provided in primary circuit 6a. Second drive circuit 624 is provided in patient circuit 6b that is electrically insulated from primary circuit 6a.

[0114] The first illumination device 603 irradiates the subject with light in the wavelength band of visible light (hereinafter simply referred to as "visible light") as first illumination light for illuminating the subject via the endoscope 201. Here, visible light refers to white light (wavelength band λ=380 nm to 780 nm). The first illumination device 603 is configured using, for example, a white LED (Light Emitting Diode) lamp or a halogen lamp.

[0115] The second illumination device 604 irradiates the subject with light in a wavelength band outside visible light (hereinafter simply referred to as "invisible light") as second illumination light for illuminating the subject via the endoscope 201. Here, the invisible light is infrared light (wavelength band λ=800 nm to 2500 nm). The second illumination device 604 is configured using, for example, an infrared LED lamp or the like.

[0116] The input unit 605 receives input of signals from each device constituting the treatment system 1, and outputs the received signals to the CPU 606 and the CPU 610 in the lighting circuit.

[0117] The CPU 606 and the lighting circuit CPU 610 work together to comprehensively control the operation of the lighting device 6. The CPU 606 reads a program stored in the memory 607 into a working area of the memory and executes it, and controls each component part through the execution of the program by the processor, whereby the hardware and software work together to control the operation of each part of the lighting device 6.

[0118] The memory 607 stores various information necessary for the operation of the lighting device 6 and various programs executed by the lighting device 6. The memory 607 is configured using RAM, ROM, and the like.

[0119] The wireless communication unit 608 is an interface for wireless communication with other devices, and is configured using a communication module capable of Wi-Fi, Bluetooth, or the like.

[0120] The communication interface 609 is an interface for communicating with the lighting circuit 6c.

[0121] The lighting circuit memory 630 stores various information and programs required for the operation of the first lighting device 603 and the second lighting device 604. The lighting circuit memory 630 is configured using RAM, ROM, and the like.

[0122] In the lighting device 6 configured in this manner, the input unit 605, CPU 606, memory 607, wireless communication unit 608, and communication interface 609 are provided in the primary circuit 6a. The first lighting device 603, the second lighting device 604, the lighting circuit CPU 610, and the lighting circuit memory 61A are provided in the lighting circuit 6c.

[0123] [Configuration of image sensor] Next, the configuration of the above-mentioned image sensor 2241 will be described. FIG. 10 is a block diagram showing the functional configuration of the image sensor 2241. 10 is realized using a CCD or CMOS image sensor having a plurality of pixels arranged in a two-dimensional matrix. Under the control of the CPU 242, the image sensor 2241 performs photoelectric conversion on a subject image (light rays) formed by an optical system (not shown) to generate image data (RAW data), and outputs this image data to the endoscope control device 202. The image sensor 2241 has a pixel unit 2241a and a color filter 2241b.

[0124] First, the configuration of the pixel portion 2241a will be described. FIG. 11 is a diagram schematically showing the configuration of the pixel section 2241a. As shown in FIG. 11, the pixel section 2241a includes a plurality of pixels P such as photodiodes that accumulate electric charges according to the amount of light. nm (n=an integer equal to or greater than 1, m=an integer equal to or greater than 1) are arranged in a two-dimensional matrix. nm Pixels P in the readout area arbitrarily set as readout targetsnm The image signal is read out as image data from the endoscope control device 202 and output to the endoscope control device 202.

[0125] Next, the configuration of the color filter 2241b will be described. FIG. 12 is a diagram showing a schematic configuration of the color filter 2241b. As shown in FIG. 12, the color filter 2241b has a basic unit (RGGB) of a Bayer array constructed using a filter R that transmits light in the red wavelength band, two filters G that transmit light in the green wavelength band, and a filter B that transmits light in the blue wavelength band, and an IR unit (RGBIR) constructed by replacing one filter G in the Bayer array with a filter IR that transmits light in the infrared wavelength band.

[0126] In the color filter 2241b configured in this manner, the basic units and the IR units are arranged at predetermined intervals. Specifically, in the color filter 2241b, the basic units and the IR units are arranged alternately with respect to the pixel portion 2241a.

[0127] Furthermore, the color filter 2241b does not need to be limited to a configuration in which basic units and IR units are arranged alternately, and can be modified as appropriate, for example, in which one IR unit is arranged for every three basic units (3:1 spacing).

[0128] [Sensitivity characteristics of each filter] Next, the sensitivity characteristics of each filter will be described. FIG. 13 is a diagram showing the sensitivity and wavelength band of each filter. In Fig. 13, the horizontal axis indicates wavelength (nm) and the vertical axis indicates transmission characteristics (sensitivity characteristics). B represents the transmission characteristics of filter B, and curve L G represents the transmission characteristic of filter G, and curve L R represents the transmission characteristic of filter R, and curve L IR indicates the transmission characteristics of the filter IR.

[0129] Curve L in Figure 13 B As shown in Fig. 13, filter B transmits light in the blue wavelength band (400 nm to 500 nm). G As shown in the curve L in FIG. 13, the filter G transmits light in the green wavelength band (480 nm to 600 nm). R As shown in the curve L in FIG. 13, the filter R transmits light in the red wavelength band (570 nm to 680 nm). IR As shown in the figure, the filter IR transmits light in the infrared wavelength band (870 nm to 1080 nm). In the following, the filter R is referred to as a pixel P nm The pixel P is an R pixel and the filter G is arranged on the light receiving surface. nm The pixel P is a G pixel, and the filter B is placed on the light receiving surface. nm The pixel P is a B pixel, and the filter IR is placed on the light receiving surface. nm will be described as an IR pixel.

[0130] [Detailed functional configuration of the image processing unit] Next, the detailed functional configuration of the image processing unit 222 will be described. FIG. 14 is a block diagram showing the detailed functional configuration of the image processing unit 222. The image processing unit 222 shown in Figure 14 has an image data input unit 2221, a first image generation unit 2222, a first detection unit 2223, a second image generation unit 2224, a second detection unit 2225, a first corrected image generation unit 2226, a second corrected image generation unit 2227, a composite image generation unit 2228, a display image generation unit 2229, a turbidity determination unit 2230, a memory 2231, and an image processing control unit 2232.

[0131] The image data input unit 2221 receives input of image data generated by the endoscope 201 and input of signals from each device constituting the treatment system 1, and outputs the received data and signals to the bus.

[0132] The first image generation unit 2222 performs predetermined image processing on image data (RAW data) input via the image data input unit 2221 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204 to generate first image data, and outputs this first image data to the first detection unit 2223, the first corrected image generation unit 2226, and the composite image generation unit 2228. Specifically, the first image generation unit 2222 generates first image data (normal color image data) based on pixel values of R pixels, G pixels, and B pixels included in the image data. Here, examples of predetermined image processing include demosaicing processing, color correction processing, black level correction processing, noise reduction processing, and gamma correction processing. In this case, the first image generation unit 2222 generates the first image data by interpolating the pixel values of IR pixels using pixel values of surrounding pixels, for example, adjacent G pixels. Note that the first image generation unit 2222 may also perform demosaicing processing by interpolating the pixel values of IR pixels using other well-known techniques, or pixel defect correction processing of color image data. In the first embodiment, the first image generation unit 2222 functions as a first image acquisition unit that acquires a first image including a region where a living body is to be treated by an energy treatment tool, for example, the ultrasonic probe 312. The first image generation unit 2222 may generate the first image data based on a drive signal of the treatment tool 301.

[0133] The first detection unit 2223 detects a change in gradation from at least a partial region of the first image (hereinafter simply referred to as "first image") corresponding to the first image data based on the first image data generated by the first image generation unit 2222, and outputs this detection result to the first corrected image generation unit 2226, the composite image generation unit 2228, and the image processing control unit 2232. Specifically, the first detection unit 2223 detects turbidity in the field of view of the endoscope 201 as at least a partial region of the first image based on the first image generated by the first image generation unit 2222, and outputs this detection result to the first corrected image generation unit 2226, the composite image generation unit 2228, and the image processing control unit 2232. The method of detecting turbidity by the first detection unit 2223 is the same as the method of detecting a turbidity component by the turbidity estimation unit 2226a of the first corrected image generation unit 2226, which will be described later, and therefore a detailed description of the detection method will be omitted.

[0134] Here, the turbidity of the field of view of the endoscope 201 refers to the degree of turbidity due to bone powder and debris dissolved in the perfusion fluid, which is a factor that degrades the gradation in the first image. Factors that degrade image quality include the dissolution of biological tissues, such as bone powder, debris, blood, and bone marrow, in the perfusion fluid, as well as smoke and sparks during treatment with the treatment tool 301. In the following, turbidity will be described as a state in which the perfusion fluid becomes cloudy when bone powder dissolves in the perfusion fluid. The perfusion fluid in which biological tissues have been dissolved becomes cloudy overall, and is characterized by high brightness, low saturation (low color reproduction), and low contrast. For this reason, the first detection unit 2223 detects the turbidity (turbidity component) of the field of view of the endoscope 201 by calculating the contrast, brightness, and saturation for each pixel that constitutes the first image as the turbidity of the field of view of the endoscope 201.

[0135] The second image generation unit 2224 performs predetermined image processing on image data (RAW data) input via the image data input unit 2221 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204 to generate second image data, and outputs this second image data to the second detection unit 2225, the second corrected image generation unit 2227, and the composite image generation unit 2228. Specifically, the second image generation unit 2224 generates second image data (infrared image data) based on pixel values of IR pixels included in the image data. Here, examples of predetermined image processing include demosaicing processing, color correction processing, black level correction processing, noise reduction processing, and gamma correction processing. In this case, the second image generation unit 2224 generates the second image data by interpolating the pixel value of the IR pixel at the target pixel and the pixel values of IR pixels surrounding the IR pixel at the target pixel. Note that the second image generation unit 2224 may interpolate the pixel values of the IR pixels using other well-known techniques. In the first embodiment, the second image generation unit 2224 functions as a second image acquisition unit that acquires second image data having a wavelength different from that of the first image. The second image generation unit 2224 may generate the second image data based on a drive signal of the treatment tool 301.

[0136] The second detection unit 2225 detects edge components from at least a partial region of a second image (hereinafter simply referred to as "second image") corresponding to the second image data based on the second image data generated by the second image generation unit 2224, and outputs this detection result to the second corrected image generation unit 2227, the composite image generation unit 2228, and the image processing control unit 2232. Specifically, the second detection unit 2225 detects edge components from a region including the endoscope 201 as at least a partial region of the second image based on the second image (infrared image) generated by the second image generation unit 2224, and outputs this detection result to the second corrected image generation unit 2227, the composite image generation unit 2228, and the image processing control unit 2232. The second detection unit 2225 detects edge components from the second image by, for example, well-known edge extraction processing. The second detection unit 2225 may also detect a change in gradation from at least a partial region of the second image by a method similar to that of the first detection unit 2223.

[0137] The first corrected image generation unit 2226 performs gradation correction on the first image input from the first image generation unit 2222 based on the detection result input from the first detection unit 2223 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204, to generate first corrected image data, and outputs the first corrected image corresponding to this first corrected image data (hereinafter simply referred to as "first corrected image") to the composite image generation unit 2228 or the display image generation unit 2229. Specifically, the first corrected image generation unit 2226 generates a first corrected image from which a cause of deterioration in visibility due to turbidity (turbidity component) included in the first image has been removed, and outputs this first corrected image to the composite image generation unit 2228 or the display image generation unit 2229. Details of the first corrected image generation unit 2226 will be described later.

[0138] The second corrected image generation unit 2227 performs gradation correction on the second image input from the second image generation unit 2224 based on the detection result input from the second detection unit 2225 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204 to generate second corrected image data, and outputs this second corrected image data (hereinafter simply referred to as "second corrected image") to the composite image generation unit 2228 or the display image generation unit 2229. Specifically, the second corrected image generation unit 2227 performs edge extraction processing on the second image to extract edge components that deteriorate visibility due to turbidity (turbid components), and generates a second corrected image by performing edge enhancement processing on the extracted edge components to enhance the edges.

[0139] Under the control of the image processing control unit 2232, the composite image generation unit 2228 generates composite image data by combining the first corrected image input from the first corrected image generation unit 2226 and the second corrected image input from the second corrected image generation unit 2227 at a predetermined ratio, and outputs a composite image corresponding to this composite image data (hereinafter simply referred to as a "composite image") to the display image generation unit 2229. Here, the predetermined ratio is, for example, 5:5. The composite image generation unit 2228 may change the ratio at which the first corrected image and the second corrected image are combined based on the ratio between the detection result of the first detection unit 2223 and the detection result of the second detection unit 2225, or may change it as appropriate, and may change the combination ratio at which the first corrected image and the second corrected image are combined depending on the component and type of turbidity. The composite image generation unit 2228 may generate a composite image by adding the edge component extracted from the second corrected image by the second detection unit 2225 to the first corrected image.

[0140] The display image generation unit 2229, under the control of the image processing control unit 2232 and in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204, generates a display image corresponding to display image data to be displayed on the display device 203 based on one or more of the first image input from the first image generation unit 2222, the second image input from the second image generation unit 2224, the first corrected image input from the first corrected image generation unit 2226, the second corrected image input from the second corrected image generation unit 2227, and the composite image input from the composite image generation unit 2228, and outputs the generated display image to the display device 203. Specifically, the display image generation unit 2229 converts, for example, the format of the input image into a predetermined format, for example, converting the RGB format into the YCbCr format, and outputs the converted image to the display device 203. The display image generated by the display image generation unit 2229 includes temporally continuous images in the field of view of the endoscope 201. The display image generating unit 2229 may generate a display image based on a drive signal of the treatment tool 301.

[0141] The turbidity determination unit 2230 determines whether the turbidity detected by the first detection unit 2223 is equal to or greater than a predetermined value, and outputs the determination result to the image processing control unit 2232. Here, the predetermined value is, for example, a value at a level where the treatment site is obscured in the field of view of the endoscope 201 due to turbidity. For example, the value at the level where the treatment site is obscured is a value of high brightness and low saturation (high brightness white).

[0142] The memory 2231 stores various types of information necessary for the operation of the image processing unit 222, various programs executed by the image processing unit 222, various types of image data, etc. The memory 2231 is configured using RAM, ROM, frame memory, etc.

[0143] The image processing control unit 2232 controls each unit constituting the image processing unit 222. The image processing control unit 2232 reads a program stored in the memory 2231 into a working area of the memory and executes it, and controls each component unit etc. through the execution of the program by the processor, thereby controlling the operation of each unit constituting the image processing unit 222 through cooperation between hardware and software.

[0144] [Detailed functional configuration of the first corrected image generation unit] Next, the detailed mechanical configuration of the first corrected image generating unit 2226 will be described. FIG. 15 is a block diagram showing the detailed functional configuration of the first corrected image generating unit 2226. As shown in FIG. The first corrected image generating unit 2226 shown in FIG. 15 includes a muddyness estimating unit 2226a, a histogram generating unit 2226b, a statistical information calculating unit 2226c, a correction coefficient calculating unit 2226d, and a contrast correcting unit 2226e.

[0145] The turbidity estimation unit 2226a estimates the turbidity component for each pixel in the first image. Here, the turbidity component for each pixel refers to the degree of turbidity due to bone powder and debris dissolved in the perfusion fluid, which is a factor in degrading the gradation in the first image. Factors that degrade image quality include phenomena caused by the dissolution of biological tissues such as bone powder, debris, blood, and bone marrow in the perfusion fluid, as well as smoke and sparks during treatment with the treatment tool 301. The following describes the turbidity of the perfusion fluid when bone powder dissolves. Perfusion fluid in which biological tissues have been dissolved is characterized by high brightness, low saturation (low color reproduction), and low contrast.

[0146] For this reason, the turbidity estimation unit 2226a calculates the contrast or brightness and saturation of the first image to estimate the turbidity component of the field of view of the endoscope 201. Specifically, the turbidity estimation unit 2226a estimates the turbidity component H(x, y) based on the R value, G value, and B value of the pixel at the coordinates (x, y) in the first image.

[0147] Here, if the R value, G value, and B value at coordinates (x, y) are Ir, Ig, and Ib, respectively, the turbidity component H(x, y) of the pixel at coordinates (x, y) is estimated by the following equation (1). H(x,y)=min(Ir,Ig,Ib) ···(1)

[0148] The turbidity estimation unit 2226a performs the calculation of the above-mentioned formula (1) for each pixel of the first image. The turbidity estimation unit 2226a sets a scan area F (small area) of a predetermined size for the first image. The size of this scan area F is, for example, a predetermined size of m×n pixels (m and n are natural numbers). In the following description, the pixel at the center of the scan area F will be referred to as the reference pixel. Furthermore, in the following description, each pixel around the reference pixel in the scan area F will be referred to as a neighboring pixel. Furthermore, in the following description, the scan area F will be described as being formed to a size of, for example, 5×5 pixels. Of course, the scan area F can also be applied if it is just one pixel.

[0149] The turbidity estimation unit 2226a calculates (Ir, Ig, Ib) for each pixel in the scan area F while shifting the position of the scan area F relative to the first image, and estimates the minimum value among them as the turbidity component H(x, y) of the reference pixel. Pixel values in high-brightness, low-saturation areas in the first image have similar and large R, G, and B values, so the value of min(Ir, Ig, Ib) is large. That is, areas with high brightness and low saturation have a large value for the turbidity component H(x, y).

[0150] In contrast, pixel values in low brightness or high saturation areas have small R, G, or B values, so the value of min(Ir, Ig, Ib) is small. That is, in low brightness or high saturation areas, the turbidity component H(x, y) has a small value.

[0151] Thus, the turbidity component H(x,y) increases as the concentration of bone powder dissolved in the perfusion fluid increases (as the white color of the bone powder increases), and decreases as the concentration of bone powder dissolved in the perfusion fluid decreases. In other words, the turbidity component H(x,y) increases as the color (white) of the perfusion fluid increases due to the bone powder dissolved in the perfusion fluid, and decreases as the color of the perfusion fluid decreases.

[0152] The turbidity estimation unit 2226a estimates the turbidity component H(x, y) using the above-mentioned formula (1), but is not limited to this, and any index showing high brightness and low saturation can be used as the turbidity component. The turbidity estimation unit 2226a may estimate the turbidity component using one or more of the local contrast value, edge intensity, color density, and object distance. Furthermore, the above-mentioned first detection unit 2223 and second detection unit 2225 detect turbidity (turbidity component) using the same method as the turbidity estimation unit 2226a.

[0153] The local histogram generation unit 2226b determines the distribution of a histogram in a local region including a reference pixel of the first image and neighboring pixels around the reference pixel, based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a. The degree of change in this turbidity component (x, y) serves as an index for determining the region to which each pixel belongs in the local region. Specifically, the degree of change in this turbidity component (x, y) is determined based on the difference in the turbidity component H(x, y) between the reference pixel and neighboring pixels in the local region.

[0154] That is, the local histogram generation unit 2226b generates a brightness histogram for a local region including neighboring pixels for each reference pixel based on the first image input from the first image generation unit 2222 and the turbidity component H(x, y) input from the turbidity estimation unit 2226a. A general histogram is generated by regarding pixel values in the target local region as brightness values and counting the frequency of pixel values one by one.

[0155] In contrast, the local histogram generation unit 2226b according to the first embodiment weights the count value for the pixel value of a neighboring pixel according to the turbidity component H(x,y) between the reference pixel and the neighboring pixel in the local region. The count value for the pixel value of the neighboring pixel is, for example, a value in the range of 0.0 to 1.0. The count value is set so that the larger the difference between the turbidity component H(x,y) between the reference pixel and the neighboring pixel is, the smaller the value becomes, and the smaller the difference between the turbidity component H(x,y) between the reference pixel and the neighboring pixel is, the larger the value becomes. Furthermore, the local region is formed to have a size of, for example, 7x7 pixels.

[0156] In general, if a histogram is generated based only on luminance, the luminance of neighboring pixels with large differences from the luminance of the pixel of interest will also be counted. Therefore, it is desirable to generate a local histogram based on the image region to which the pixel of interest belongs.

[0157] In contrast, in generating a brightness histogram in the first embodiment, a count value for the pixel value of each pixel in a local region in the first image data is set according to the difference in the turbidity component H(x,y) between a reference pixel and each neighboring pixel in the local region in the first image data. Specifically, the count value is calculated using, for example, a Gaussian function so that the larger the difference in the turbidity component H(x,y) between the reference pixel and the neighboring pixels, the smaller the count value, and the smaller the difference in the turbidity component H(x,y) between the reference pixel and the neighboring pixels, the larger the count value (see, for example, Japanese Patent No. 6720012 or Japanese Patent No. 6559229, where the haze component is replaced with the turbidity component).

[0158] The method of calculating the count value by the local histogram generation unit 2226b is not limited to a Gaussian function, and any method may be used as long as the count value decreases as the difference between the values of the reference pixel and the neighboring pixels increases. For example, the local histogram generation unit 2226b may calculate the count value using a lookup table or a table approximated by a broken line instead of a Gaussian function.

[0159] The local histogram generating unit 2226b may also compare the difference in value between the reference pixel and the neighboring pixel with a threshold, and if the difference is equal to or greater than the threshold, reduce the count value of the neighboring pixel (for example, set it to 0.0).

[0160] Furthermore, the local histogram generation unit 2226b does not necessarily have to use the frequency of pixel values as count values. For example, the local histogram generation unit 2226b may use each of the R value, G value, and B value as count values. Furthermore, the local histogram generation unit 2226b may use the G value as a luminance value as count value.

[0161] The statistical information calculation unit 2226c calculates a representative luminance based on the statistical information of the luminance histogram input from the local histogram generation unit 2226b. The representative luminance is the luminance of the low luminance part, the luminance of the high luminance part, and the luminance of the intermediate luminance part of the effective luminance range of the luminance histogram. The luminance of the low luminance part is the minimum luminance of the effective luminance range. The luminance of the high luminance part is the maximum luminance of the effective luminance range. The luminance of the intermediate luminance part is the centroid luminance. The minimum luminance is the luminance at which the cumulative frequency is 5% of the maximum value in a cumulative histogram created from the luminance histogram. The maximum luminance is the luminance at which the cumulative frequency is 95% of the maximum value in a cumulative histogram created from the luminance histogram. The centroid luminance is the luminance at which the cumulative frequency is 50% of the maximum value in a cumulative histogram created from the luminance histogram.

[0162] The percentages of cumulative frequency corresponding to the minimum luminance, maximum luminance, and centroid luminance, 5%, 50%, and 95%, can be changed as appropriate. Furthermore, the luminance of the intermediate luminance portion is set to the centroid luminance in the cumulative histogram, but this is not limited thereto, and the centroid luminance does not necessarily have to be calculated from the cumulative frequency. For example, the luminance of the intermediate luminance portion can also be set to the luminance with the highest frequency in the luminance histogram.

[0163] The correction coefficient calculation unit 2226d calculates a correction coefficient for correcting the contrast in a local region based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the statistical information input from the statistical information calculation unit 2226c. Specifically, when contrast correction is performed by histogram expansion, the correction coefficient calculation unit 2226d calculates a coefficient for histogram expansion using the centroid luminance and maximum luminance from the statistical information.

[0164] Here, histogram expansion is a process that enhances contrast by expanding the effective luminance range of the histogram (see, for example, Japanese Patent No. 6720012 or Japanese Patent No. 6559229). Note that the correction coefficient calculation unit 2226d uses histogram expansion as a means of achieving contrast correction, but is not limited to this. Histogram flattening, for example, may also be applied as a means of achieving contrast correction. For example, the correction coefficient calculation unit 2226d may use a cumulative histogram or a table that approximates a broken line as a method of achieving histogram flattening. This cumulative histogram is obtained by sequentially accumulating the most frequent values of the luminance histogram.

[0165] The contrast correction unit 2226e performs contrast correction of the reference pixel of the first image data for the first image input from the first image generation unit 2222 based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the correction coefficient input from the correction coefficient calculation unit 2226d (see, for example, Patent Publication No. 6720012 or Patent Publication No. 6559229).

[0166] The first corrected image generation unit 2226 configured in this way estimates the turbidity component H(x, y) based on the first image, calculates a brightness histogram and representative brightness using this estimation result, calculates a correction coefficient for correcting contrast within a local region, and performs contrast correction based on the turbidity component H(x, y) and the correction coefficient. In this way, the first corrected image generation unit 2226 can generate a first corrected image in which turbidity has been removed from the first image.

[0167] [Summary of treatment] Next, an outline of the treatment performed by the surgeon using the treatment system 1 will be described. 16 is a flowchart outlining a procedure performed by an operator using the treatment system 1. The operator performing the procedure may be a single doctor, or may be two or more people including a doctor and an assistant.

[0168] As shown in FIG. 16, first, the surgeon forms a first portal P1 and a second portal P2 that respectively connect the inside of the joint cavity C1 of the knee joint J1 with the outside of the skin (step S1).

[0169] Next, the surgeon inserts the endoscope 201 into the joint cavity C1 through the first portal P1, inserts the guiding device 4 into the joint cavity C1 through the second portal P2, and inserts the treatment tool 301 into the joint cavity C1 under the guidance of the guiding device 4 (step S2). Note that, although the case where two portals are formed and then the endoscope 201 and the treatment tool 301 are inserted into the joint cavity C1 through the first portal P1 and the second portal P2 has been described here, it is also possible to form the first portal P1 and insert the endoscope 201 into the joint cavity C1, and then form the second portal P2 and insert the guiding device 4 and the treatment tool 301 into the joint cavity C1.

[0170] Thereafter, the surgeon brings the ultrasonic probe 312 into contact with the bone to be treated while visually checking the endoscopic image of the inside of the joint cavity C1 displayed on the display device 203 (step S3).

[0171] Next, the surgeon performs ablation treatment using the treatment tool 301 while viewing the endoscopic image displayed on the display device 203 (step S4). Details of the processing of the treatment system 1 during ablation treatment will be described later.

[0172] Thereafter, the display device 203 performs a display and notification process for displaying information about the inside of the joint cavity C1 and the state after the cutting treatment (step S5). The endoscope control device 202 stops the display and notification after a predetermined time has elapsed after the display and notification process. The surgeon then finishes the treatment using the treatment system 1.

[0173] [Details of cutting procedure] Next, the cutting treatment in step S4 of FIG. 16 will be described in detail. FIG. 17 provides an overview of the processing executed by the endoscope control device 202 during cutting treatment. In the following description, each process is executed under the control of the CPU of each control device, but the processes may be executed collectively by one of the control devices such as the network control device 7, for example.

[0174] The CPU 227 communicates with each device, sets control parameters for each of the treatment device 3 and the perfusion device 5, and inputs the control parameters for each of the treatment device 3 and the perfusion device 5 (step S11).

[0175] Next, the CPU 227 determines whether or not the devices of the parts constituting the treatment system 1 have entered an output ON state (step S12). If the CPU 227 determines that the devices of the parts constituting the treatment system 1 have entered an output ON state (step S12: Yes), the endoscope control device 202 proceeds to step S13, which will be described later. On the other hand, if the CPU 227 determines that the devices of the parts constituting the treatment system 1 have not entered an output ON state (step S12: No), the CPU 227 continues this determination until the devices of the parts constituting the treatment system 1 enter an output ON state.

[0176] In step S13, the CPU 227 determines whether the observation mode of the endoscope control device 202 in the treatment system 1 is set to the turbidity detection mode. If the CPU 227 determines that the observation mode of the endoscope control device 202 in the treatment system 1 is set to the turbidity detection mode (step S13: Yes), the endoscope control device 202 proceeds to step S14, which will be described later. On the other hand, if the CPU 227 determines that the observation mode of the endoscope control device 202 in the treatment system 1 is not set to the turbidity detection mode (step S13: No), the endoscope control device 202 proceeds to step S16, which will be described later.

[0177] In step S14, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 based on any one of the first image generated by the endoscope 201, the detection result of the impedance detection unit 330 of the treatment tool control device 302, and the detection result of the turbidity detection unit 516 of the perfusion device 5. Specifically, when the first image generated by the endoscope 201 is used, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 using either the brightness or the contrast of the first image. Furthermore, when the impedance detected by the impedance detection unit 330 of the treatment tool control device 302 is used, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 based on the rate of change of the impedance. Furthermore, when the detection result of the turbidity detection unit 516 of the perfusion device 5 is used, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 based on the turbidity of the perfusion fluid detected by the turbidity detection unit 516 of the perfusion device 5.

[0178] Next, the CPU 227 determines whether or not the turbidity in the field of view of the endoscope 201 is equal to or greater than a predetermined value based on the detection result detected by the turbidity detection unit 223 (step S15).

[0179] Specifically, when the turbidity detection unit 223 uses the first image, the CPU 227 determines whether the average value of the sum of the brightness values of each pixel of the first image detected by the turbidity detection unit 223 is equal to or greater than a predetermined value. Here, the predetermined value for brightness is a high luminance value that is as close to white as possible. In this case, the CPU 227 determines that turbidity has occurred in the field of view of the endoscope 201 when the average value of the sum of the brightness values of each pixel of the first image detected by the turbidity detection unit 223 is equal to or greater than the predetermined value. On the other hand, the CPU 227 determines that turbidity has not occurred in the field of view of the endoscope 201 when the average value of the sum of the brightness and saturation of each pixel of the first image detected by the turbidity detection unit 223 is not equal to or greater than the predetermined value.

[0180] Furthermore, when the turbidity detection unit 223 uses the impedance detected by the impedance detection unit 330, the CPU 227 determines whether the impedance is equal to or greater than a predetermined value. When the impedance detected by the impedance detection unit 330 by the turbidity detection unit 223 is equal to or greater than a predetermined value, the CPU 227 determines that turbidity has occurred in the field of view of the endoscope 201. On the other hand, when the impedance detected by the impedance detection unit 330 is not equal to or greater than the predetermined value, the turbidity detection unit 223 determines that turbidity has not occurred in the field of view of the endoscope 201.

[0181] Furthermore, when the turbidity detection unit 223 uses the turbidity of the perfusion fluid detected by the turbidity detection unit 516 of the perfusion device 5, the CPU 227 determines whether the turbidity of the perfusion fluid is equal to or greater than a predetermined value. When the turbidity of the perfusion fluid detected by the turbidity detection unit 223 is equal to or greater than the predetermined value, the CPU 227 determines that turbidity has occurred in the field of view of the endoscope 201. On the other hand, when the turbidity of the perfusion fluid detected by the turbidity detection unit 223 is not equal to or greater than the predetermined value, the CPU 227 determines that turbidity has not occurred in the field of view of the endoscope 201.

[0182] In step S15, if the CPU 227 determines that the field of view of the endoscope 201 is cloudy (step S15: Yes), the endoscope control device 202 proceeds to step S19, which will be described later. On the other hand, if the CPU 227 determines that the field of view of the endoscope 201 is not cloudy (step S15: No), the endoscope control device 202 proceeds to step S16, which will be described later.

[0183] In step S16, the CPU 227 performs normal control over the endoscope control device 202. Specifically, the CPU 227 outputs and displays the first image (color image) generated by the image processing unit 222 on the display device 203. This allows the surgeon to perform treatment using the treatment tool 301 while viewing the first image displayed on the display device 203, even when the field of view around the treatment area is blurred.

[0184] Next, the CPU 227 determines whether the surgeon is continuing the treatment on the subject (step S17). Specifically, the CPU 227 determines whether the treatment instrument control device 302 is supplying power to the treatment instrument 301. If the treatment instrument control device 302 is supplying power to the treatment instrument 301, the CPU 227 determines that the surgeon is continuing the treatment on the subject. If the treatment instrument control device 302 is not supplying power to the treatment instrument 301, the CPU 227 determines that the surgeon is not continuing the treatment on the subject. If the CPU 227 determines that the surgeon is continuing the treatment on the subject (step S17: Yes), the endoscope control device 202 proceeds to step S18, which will be described later. On the other hand, if the CPU 227 determines that the surgeon is not continuing the treatment on the subject (step S17: No), the endoscope control device 202 ends this process.

[0185] In step S18, the CPU 227 determines whether or not the output of each device constituting the treatment system 1 has been turned off. If the CPU 227 determines that the output of each device constituting the treatment system 1 has been turned off (step S18: Yes), the endoscope control device 202 ends this processing. On the other hand, if the CPU 227 determines that the output of each device constituting the treatment system 1 has not been turned off (step S18: No), the endoscope control device 202 returns to step S13 described above.

[0186] In step S19, the endoscope control device 202 executes turbidity response control processing for turbidity in the field of view of the endoscope 201. Details of the turbidity response control processing will be described later. After step S19, the endoscope control device 202 proceeds to step S17.

[0187] [Details of Turbidity Control Processing] Next, a description will be given of the details of the turbidity response control process described in step S19 of Fig. 17. Fig. 18 is a flowchart showing a detailed outline of the turbidity response control process of Fig. 17.

[0188] 18, first, the image processing unit 222 generates a first image and a second image (step S101). Specifically, the first image generation unit 2222 generates a first image (a color image using visible light) based on image data input from the image data input unit 2221. Furthermore, the second image generation unit 2224 generates a second image (an IR image using invisible light) based on the image data input from the image data input unit 2221.

[0189] Next, the second-corrected image generation unit 2227 performs a well-known edge enhancement process on the second image (step S102). Specifically, the second-corrected image generation unit 2227 performs edge extraction to extract areas of the second image where brightness changes significantly, and then performs edge enhancement to enhance the edges of the extracted areas. The edge enhancement process by the second-corrected image generation unit 2227 may be performed by combining well-known dilation, erosion, averaging, and median processes, for example. The edge extraction may be performed by combining one or more of well-known Sobel filters, Laplacian filters, and Canny filters, for example.

[0190] Thereafter, the first detection unit 2223 estimates the turbidity component in the field of view of the endoscope 201 based on the first image generated by the first image generation unit 2222 (step S103). Specifically, the first detection unit 2223 estimates the turbidity component in the field of view of the endoscope 201 using the same estimation method as that used by the turbidity estimation unit 2226a described above.

[0191] Next, the turbidity determination unit 2230 determines whether the turbidity of the field of view of the endoscope 201 detected by the first detection unit 2223 is equal to or greater than a predetermined value. If the turbidity determination unit 2230 determines that the turbidity component of the field of view of the endoscope 201 detected by the first detection unit 2223 is equal to or greater than a predetermined value (step S104: Yes), the endoscope control device 202 proceeds to step S105, which will be described later. On the other hand, if the turbidity determination unit 2230 determines that the turbidity component of the field of view of the endoscope 201 detected by the first detection unit 2223 is not equal to or greater than the predetermined value (step S104: No), the endoscope control device 202 proceeds to step S114, which will be described later.

[0192] In step S105, the first corrected image generation unit 2226 performs a muddy correction process on the first image to remove or reduce muddiness. Specifically, first, the muddy estimation unit 2226a estimates a muddy component H(x,y) for the first image. Next, the local histogram generation unit 2226b determines a histogram distribution in a local region including a reference pixel of the first image and neighboring pixels around the reference pixel based on the muddy component H(x,y) input from the muddy estimation unit 2226a. After that, the statistical information calculation unit 2226c calculates a representative luminance based on statistical information of the luminance histogram input from the local histogram generation unit 2226b. Next, the correction coefficient calculation unit 2226d calculates a correction coefficient for correcting the contrast in the local region based on the muddy component H(x,y) input from the muddy estimation unit 2226a and the statistical information input from the statistical information calculation unit 2226c. Finally, the contrast correction unit 2226e performs contrast correction of the reference pixel of the first image input from the first image generation unit 2222 based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the correction coefficient input from the correction coefficient calculation unit 2226d.

[0193] The image processing control unit 2232 determines whether the display mode of the endoscope control device 202 is set to a correction mode in which an image in which the turbidity component has been corrected is displayed (step S106). If the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is set to the correction mode in which an image in which the turbidity component has been corrected is displayed (step S106: Yes), the endoscope control device 202 proceeds to step S107, which will be described later. On the other hand, if the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is not set to the correction mode in which an image in which the turbidity component has been corrected is displayed (step S106: No), the endoscope control device 202 proceeds to step S108, which will be described later.

[0194] In step S107, the display image generation unit 2229 generates a first corrected image based on the first image in which the turbidity has been corrected by the first corrected image generation unit 2226, and outputs the first corrected image to the display device 203. After step S107, the endoscope control device 202 returns to the main routine of the cutting procedure in Fig. 17 described above, and proceeds to step S17.

[0195] Fig. 19 is a diagram showing an example of a first image that the display image generation unit 2229 generates based on the first image and outputs to the display device 203 when the muddiness correction process by the first corrected image generation unit 2226 has not been performed. Fig. 20 is a diagram showing an example of a first corrected image that the display image generation unit 2229 generates based on the first corrected image and outputs to the display device 203 when the muddiness correction process by the first corrected image generation unit 2226 has been performed. Note that the time axes in Fig. 19 and Fig. 20 are the same.

[0196] As shown in display images P1 to P5 of Figure 19, when the field of view of the endoscope 201 becomes cloudy due to bone powder or the like dissolving in the perfusion fluid when the treatment target area 100 is treated with the ultrasonic vibrator 312a of the ultrasonic probe 312, the surgeon cannot confirm the positions of the ultrasonic vibrator 312a of the ultrasonic probe 312 and the treatment target area 100 in the field of view of the endoscope 201, nor the state of cutting or the like performed on the treatment target area 100 by the ultrasonic vibrator 312a.

[0197] 20, when the treatment of the treatment target area 100 by the ultrasonic vibrator 312a of the ultrasonic probe 312 causes the perfusion fluid to become cloudy and opaque due to dissolution of bone powder or the like in the perfusion fluid, the first corrected image generation unit 2226 outputs to the display device 203 a first corrected image in which the opacity has been reduced or removed (for example, first corrected image P13 (time t=t3) and first corrected image P14 (time t=4)). This allows the surgeon to confirm the positions of the ultrasonic vibrator 312a of the ultrasonic probe 312 and the treatment target area 100 in the field of view of the endoscope 201, and the state of cutting or the like by the ultrasonic vibrator 312a on the treatment target area 100, so that cutting of the treatment target area 100 by the ultrasonic probe 312 can be performed without interruption.

[0198] Returning to FIG. 18, the description of step S108 and subsequent steps will be continued. In step S108, the image processing control unit 2232 determines whether the display mode of the endoscope control device 202 is set to the IR mode that displays the IR image, which is the second image. If the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is set to the IR mode that displays the IR image, which is the second image (step S108: Yes), the endoscope control device 202 proceeds to step S109, which will be described later. On the other hand, if the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is not set to the IR mode that displays the IR image, which is the second image (step S108: No), the endoscope control device 202 proceeds to step S110, which will be described later.

[0199] In step S109, the display image generation unit 2229 generates a second corrected image, which is an edge-enhanced IR image, based on the second image generated by the second corrected image generation unit 2227, and outputs the second corrected image to the display device 203. After step S107, the endoscope control device 202 returns to the main routine for the cutting procedure in Fig. 17 described above, and proceeds to step S17.

[0200] 21 is a diagram showing an example of a second corrected image that is generated by the display image generation unit 2229 based on the second corrected image and output to the display device 203 when edge enhancement processing is performed by the second corrected image generation unit 2227. The time axis in FIG. 21 is the same as the time axis in FIG. 19 described above.

[0201] 21, when the field of view of the endoscope 201 becomes cloudy and opaque due to treatment of the treatment target area 100 by the ultrasonic probe 312, the display image generation unit 2229 outputs to the display device 203 second corrected images in which the second corrected image generation unit 2227 has performed edge enhancement processing to enhance the contours of the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100 (for example, second corrected image P23 (time t=t3) and second corrected image P24 (time t=4)). This allows the surgeon to indirectly confirm the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100, and therefore allows the surgeon to perform cutting of the treatment target area 100 by the ultrasonic probe 312 without interruption.

[0202] Returning to FIG. 18, the description of step S110 and subsequent steps will be continued. In step S110, the image processing control unit 2232 determines whether the display mode of the endoscope control device 202 is set to a synthesis mode in which a composite image obtained by synthesizing the first corrected image and the second corrected image is displayed. If the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is set to the synthesis mode in which a composite image obtained by synthesizing the first corrected image and the second corrected image is displayed (step S110: Yes), the endoscope control device 202 proceeds to step S111, which will be described later. On the other hand, if the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is not set to the synthesis mode in which a composite image obtained by synthesizing the first corrected image and the second corrected image is displayed (step S110: No), the endoscope control device 202 proceeds to step S112 (side-by-side display mode), which will be described later.

[0203] In step S111, the composite image generation unit 2228 generates a composite image by combining the first corrected image generated by the first corrected image generation unit 2226 and the second corrected image generated by the second corrected image generation unit 2227 at a predetermined ratio, for example, 5:5.

[0204] Next, the display image generation unit 2229 outputs the composite image generated by the composite image generation unit 2228 to the display device 203 (step S112). After step S112, the endoscope control device 202 returns to the main routine of the cutting treatment in Fig. 17 described above, and proceeds to step S17.

[0205] 22 is a diagram showing an example of a composite image that is generated by the display image generation unit 2229 based on the composite image and output to the display device 203 when the composite processing is performed by the composite image generation unit 2228. Note that the time axis in FIG. 22 is the same as the time axis in FIG. 19 described above.

[0206] 22, when the field of view of the endoscope 201 becomes cloudy due to treatment of the treatment target area 100 with the ultrasonic probe 312, the display image generation unit 2229 combines a first corrected image in which the cloudiness has been reduced or removed by the first corrected image generation unit 2226 with a second corrected image in which the second corrected image generation unit 2227 has performed edge enhancement processing to enhance the contours of the ultrasonic probe 312 and the treatment target area 100, and outputs the combined image to the display device 203 (for example, composite image P33 (time t=t3) and composite image P34 (time t=4)). As a result, the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100 are emphasized compared to other areas, allowing the surgeon to easily confirm the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100, and allowing the surgeon to perform cutting of the treatment target area 100 with the ultrasonic probe 312 without interruption.

[0207] Returning to FIG. 18, the description of step S113 and subsequent steps will be continued. In step S113, the display image generation unit 2229 outputs the first corrected image generated by the first corrected image generation unit 2226 and the second corrected image generated by the second corrected image generation unit 2227 in parallel to the display device 203. After step S113, the endoscope control device 202 returns to the main routine of the cutting treatment in Fig. 17 described above, and proceeds to step S17.

[0208] 23 is a diagram showing an example of an image in which the display image generation unit 2229 outputs the first corrected image and the second corrected image to the display device 203. The time axis in FIG. 23 is the same as the time axis in FIG. 19 described above.

[0209] 23, when the field of view of the endoscope 201 becomes cloudy due to treatment of the treatment target area 100 by the ultrasonic probe 312, the display image generation unit 2229 outputs a first corrected image generated by the first corrected image generation unit 2226 and a second corrected image generated by the second corrected image generation unit 2227 in parallel to the display device 203 (for example, first image P43, second image P53 (time t=t3), first image P44, second image P54 (time t=t4)). This allows the surgeon to perform treatment while visually comparing the state in which the cloudiness has been removed with the state in which the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100 are emphasized.

[0210] In step S114, the image processing control unit 2232 determines whether the display mode of the endoscope control device 202 is set to the IR mode that displays the second image, which is an infrared image. If the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is set to the IR mode that displays the second image, which is an infrared image (step S114: Yes), the endoscope control device 202 proceeds to step S115, which will be described later. On the other hand, if the image processing control unit 2232 determines that the display mode of the endoscope control device 202 is not set to the IR mode that displays the second image, which is an infrared image (step S114: No), the endoscope control device 202 proceeds to step S116, which will be described later.

[0211] In step S115, the display image generation unit 2229 generates a display image using the second image generated by the second image generation unit 2224 and outputs the display image to the display device 203. This allows the surgeon to treat the treatment target area 100 with the ultrasonic probe 312 while viewing the second infrared image displayed by the display device 203. After step S115, the endoscope control device 202 returns to the main routine for the cutting treatment in Fig. 17 described above and proceeds to step S17.

[0212] In step S116, the display image generation unit 2229 generates a display image using the first image generated by the first image generation unit 2222 and outputs the display image to the display device 203. This allows the surgeon to treat the treatment target area 100 with the ultrasonic probe 312 while viewing the color first image displayed on the display device 203. After step S116, the endoscope control device 202 returns to the main routine for the cutting treatment in Fig. 17 described above and proceeds to step S17.

[0213] According to the above-described embodiment 1, the display image generating unit 2229 generates a display image based on the first corrected image input from the first corrected image generating unit 2226 and outputs the display image to the display device 203. Therefore, even if the field of view of the endoscope 201 deteriorates, treatment of the treatment target area 100 by the treatment tool 301 can be continued.

[0214] Furthermore, according to the first embodiment, the display image generation unit 2229 generates a display image based on the composite image input from the composite image generation unit 2228, and outputs the display image to the display device 203. As a result, the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100 are emphasized compared to other areas, and the surgeon can easily confirm the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100, and therefore, the surgeon can perform cutting of the treatment target area 100 with the ultrasonic probe 312 without interruption.

[0215] Furthermore, according to the first embodiment, the display image generation unit 2229 generates a display image based on one or more of the first image input from the first image generation unit 2222, the second image input from the second image generation unit 2224, the first corrected image input from the first corrected image generation unit 2226, the second corrected image input from the second corrected image generation unit 2227, and the composite image input from the composite image generation unit 2228 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204, and outputs the display image to the display device 203. As a result, the surgeon can perform cutting on the treatment target site 100 with the ultrasonic probe 312 without interruption while viewing the smooth display image displayed by the display device 203.

[0216] Furthermore, according to embodiment 1, when the turbidity determination unit 2230 determines that the turbidity of the field of view of the endoscope 201 is equal to or greater than a predetermined value, the display image generation unit 2229 generates a display image based on the first corrected image input from the first corrected image generation unit 2226 and outputs it to the display device 203, whereas when the turbidity determination unit 2230 determines that the turbidity of the field of view of the endoscope 201 is not equal to or greater than the predetermined value, the display image generation unit 2229 generates a display image based on the first image generated by the first image generation unit 2222 and outputs it to the display device 203, so that a normal display image (color image) can be displayed until the field of view of the endoscope 201 becomes turbid.

[0217] In addition, in embodiment 1, the second corrected image generation unit 2227 may generate second corrected image data by performing gradation correction (e.g., edge enhancement processing) on the second image of infrared light based on the detection result of turbidity in the first image by the first detection unit 2223, and the display image generation unit 2229 may output a display image using the second corrected image data from the second corrected image generation unit 2227 to the display device 203.

[0218] Furthermore, in the first embodiment, the first corrected image generation unit 2226 may generate first corrected image data by performing gradation correction (e.g., turbidity correction processing) on the color first image based on the detection result of turbidity in the second image by the second detection unit 2225, and the display image generation unit 2229 may output a display image using the first corrected image data from the first corrected image generation unit 2226 to the display device 203.

[0219] (Embodiment 2) Next, a second embodiment will be described. In the first embodiment described above, the first color image and the second IR image are generated by one imaging unit 204, but in the second embodiment, the first color image and the second IR image are generated by two imaging units. Specifically, the configuration of the endoscope is different in the second embodiment. Therefore, the endoscope according to the second embodiment will be described below. Note that the same components as those in the treatment system 1 according to the first embodiment described above are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0220] [Functional configuration of the endoscope] FIG. 24 is a block diagram showing a functional configuration of an endoscope according to the second embodiment. The endoscope 201A shown in FIG. 24 includes a first imaging section 2242 and a second imaging section 2243 instead of the imaging section 204 of the endoscope 201 according to the first embodiment described above.

[0221] The first imaging unit 2242 is configured using multiple optical systems and a CCD or CMOS image sensor having a Bayer array color filter sensitive to visible light (wavelength band λ=380 nm to 780 nm) arranged on the light receiving surface. The first imaging unit 2242 generates a first image (RAW data from which color first image data can be generated) by capturing an object image formed by the optical system, and outputs the generated first image to the endoscope control device 202.

[0222] The second imaging unit 2243 is configured using multiple optical systems and a CCD or CMOS image sensor having an IR filter sensitive to invisible light (wavelength band λ=780 nm to 2500 nm) arranged on its light receiving surface. The second imaging unit 2243 captures the subject image formed by the optical system to generate a second image (RAW data from which second IR image data can be generated), and outputs the generated second image to the endoscope control device 202.

[0223] In a cutting procedure using the endoscope 201A configured in this manner, the endoscope control device 202 performs the same processing as in the cutting procedure according to the above-described embodiment 1. Therefore, a detailed description of the cutting procedure using the endoscope 201A will be omitted. Note that even in a cutting procedure using the endoscope 201A, the composite image generation unit 2228 can generate a composite image.

[0224] Fig. 25 is a diagram showing an example of a composite image generated by the composite image generation unit 2228. As shown in Fig. 25, the composite image generation unit 2228 generates a composite image P63 by combining, at a predetermined ratio, a first corrected image P61, which is a color first image generated by the first imaging unit 2242 and in which turbidity has been reduced or removed by the first corrected image generation unit 2226, and a second corrected image P62, which is an IR second image generated by the second imaging unit 2243 and in which edge enhancement processing has been performed by the second corrected image generation unit 2227.

[0225] The display image generation unit 2229 outputs the composite image P63 generated by the composite image generation unit 2228 to the display device 203.

[0226] This allows the surgeon to easily check the ultrasonic vibrator 312a of the ultrasonic probe 312 and the treatment area 100 with the turbidity removed or reduced, allowing the surgeon to perform cutting on the treatment area 100 using the ultrasonic probe 312 without interruption.

[0227] According to the above-described embodiment 2, the same effects as those of the above-described embodiment 1 are achieved, and even if the field of view of the endoscope 201A deteriorates, treatment of the treatment target area 100 by the treatment tool 301 can be continued.

[0228] (Embodiment 3) Next, a third embodiment will be described. In the first embodiment described above, the first illumination device 603 and the second illumination device 604 each irradiate visible light and invisible light toward the subject, but in the third embodiment, light in the red wavelength band, light in the green wavelength band, light in the blue wavelength band, and light in the infrared wavelength band are provided and irradiated toward the subject in a frame sequential manner. Specifically, the configurations of the endoscope and illumination device in the third embodiment are different. Therefore, the configurations of the endoscope and illumination device according to the third embodiment will be described below. Note that the same components as those in the treatment system 1 according to the first embodiment described above are assigned the same reference numerals, and detailed description thereof will be omitted.

[0229] [Functional configuration of the endoscope] FIG. 26 is a block diagram showing a functional configuration of an endoscope according to the third embodiment. The endoscope 201B shown in FIG. 26 includes an imaging section 2244 instead of the imaging section 204 of the endoscope 201 according to the first embodiment described above.

[0230] The imaging unit 2244 is configured using multiple optical systems and a CCD or CMOS image sensor having pixels sensitive to visible light (wavelength band λ=400 nm to 680 nm) and invisible light (wavelength band λ=870 nm to 1080 nm). The imaging unit 2244 captures an image of a subject formed by the optical systems to generate image data (RAW data) including the wavelength range of visible light or invisible light, and outputs the generated image data to the endoscope control device 202.

[0231] [Functional configuration of lighting device] FIG. 27 is a block diagram showing a functional configuration of the lighting device according to the third embodiment. The lighting device 7 shown in Figure 27 omits the second lighting device 604 and the second lighting control unit 602 from the lighting device 6 according to the above-mentioned embodiment 1, and includes a lighting unit 800 instead of the first lighting device 603.

[0232] The illumination unit 800, under the control of the first illumination control unit 601 and the CPU 610 in the illumination circuit, provides and irradiates light in the red wavelength band, light in the green wavelength band, light in the blue wavelength band, and light in the infrared wavelength band onto the subject in a frame sequential manner.

[0233] [Schematic configuration of lighting unit] FIG. 28 is a schematic diagram showing a schematic configuration of the illumination unit 800. As shown in FIG. The illumination unit 800 shown in FIG. 28 includes a light source 801 capable of emitting white light, and a rotary filter 802 that is disposed on the optical path of the white light emitted by the light source 801 and that is rotated by a drive unit (not shown).

[0234] Rotary filter 802 has red filter 802a that transmits light in the red wavelength band, green filter 802b that transmits light in the green wavelength band, blue filter 802c that transmits light in the blue wavelength band, and IR filter 802d that transmits light in the infrared wavelength band. Rotary filter 802 rotates to position one of red filter 802a, green filter 802b, blue filter 802c, and IR filter 802d on the optical path of the white light emitted by light source 801.

[0235] FIG. 29 is a diagram showing the relationship between the transmission characteristics and wavelength bands of red filter 802a, green filter 802b, and blue filter 802c. FIG. 30 is a diagram showing the relationship between the transmission characteristics and wavelength band of the IR filter 802d. 29 and 30, the horizontal axis represents wavelength and the vertical axis represents transmittance. RR represents the transmission characteristics of the red filter 802a, and the curve L GG represents the transmission characteristics of the green filter 802b, and the curve L BB represents the transmission characteristic of the blue filter 802c. Furthermore, in FIG.IRR indicates the transmission characteristics of the IR filter 802d.

[0236] As shown in Figures 29 and 30, the rotary filter 802 rotates under the drive of a drive unit (not shown), thereby transmitting light in the red wavelength band, light in the green wavelength band, light in the blue wavelength band, and light in the infrared wavelength band toward the subject.

[0237] In a resection procedure using the illumination device 7 configured in this manner, the endoscope control device 202 performs the same processing as in the resection procedure according to the first embodiment described above. Specifically, the endoscope control device 202 generates a first color image using red image data, green image data, and blue image data generated by the imaging unit 2244 sequentially receiving light in the red wavelength band, light in the green wavelength band, light in the blue wavelength band, and light in the infrared wavelength band, and generates a second infrared image using the infrared image data. In this case, the image processing unit 222 generates one or more of a first corrected image, a second corrected image, and a composite image using the first and second images, and outputs them to the display device 203. This provides the same effects as in the first embodiment described above, and allows the surgeon to easily check the ultrasonic transducer 312a of the ultrasonic probe 312 and the treatment target area 100 with turbidity removed or reduced, allowing the surgeon to perform resection of the treatment target area 100 using the ultrasonic probe 312 without interruption.

[0238] According to the above-described embodiment 3, the same effects as those of the above-described embodiment 1 are achieved, and even if the field of view of the endoscope 201B deteriorates, treatment of the treatment target area 100 by the treatment tool 301 can be continued.

[0239] In the third embodiment, the rotating filter 802 is rotated to irradiate light in the red wavelength band, light in the green wavelength band, light in the blue wavelength band, and light in the infrared wavelength band toward the subject. However, the present invention is not limited to this. For example, the configuration may be such that a red LED capable of irradiating light in the red wavelength band, a green LED capable of irradiating light in the green wavelength band, a blue LED capable of irradiating light in the blue wavelength band, and an infrared LED capable of irradiating light in the infrared wavelength band are used, and the red LED, green LED, blue LED, and infrared LED are sequentially illuminated to irradiate light.

[0240] In addition, in embodiment 3, a first rotary filter having an R filter, a G filter, and a B filter that can transmit light in the red wavelength band, light in the green wavelength band, and light in the blue wavelength band, respectively, and a second rotary filter having an IR filter that can transmit light in the infrared wavelength band may be provided, and the first rotary filter or the second rotary filter may be positioned on the optical path of the light source 801 and rotated depending on the mode set in the endoscope control device 202.

[0241] Furthermore, in the third embodiment, a rotary filter having an R filter, a G filter, a B filter, and a transparent filter that can transmit light in the red wavelength band, light in the green wavelength band, and light in the blue wavelength band, respectively, a first light source that can irradiate white light, and a second light source that can irradiate infrared light may be provided, and either the first light source or the second light source may be caused to emit light according to the mode set in the endoscope control device 202. In the case of the frame sequential type, the effective number of pixels of the image sensor can be increased, so the resolution per pixel is higher than when color filters are provided on the image sensor, enabling finer bone powder to be identified.

[0242] Furthermore, in the third embodiment, the light is emitted in a frame sequential manner, but the present invention is not limited to this, and the light may be emitted in a simultaneous manner.

[0243] (Modifications of Embodiments 1 to 3) In the above-described first to third embodiments, the display image generating unit 2229 switches the image to be output to the display device 203 in accordance with the mode set in the endoscope control device 202, but the present invention is not limited to this. For example, the display image generating unit 2229 may switch the image to be output to the display device 203 based on a drive signal and a synchronization signal (VT) for the treatment tool 301 input from the treatment tool control device 302. Specifically, when either a drive signal for driving the treatment tool 301 or a synchronization signal (VT) is input from the treatment tool control device 302, the display image generating unit 2229 outputs one or more of the first corrected image, the second corrected image, and the composite image to the display device 203.

[0244] This allows the surgeon to change the content of the display image displayed on the display device 203 without changing the mode of the endoscope control device 202 each time, allowing the surgeon to perform cutting on the treatment target area 100 using the ultrasonic probe 312 without performing complicated tasks.

[0245] Furthermore, the display image generation unit 2229 switches the type of image to be output to the display device 203 in accordance with the synchronization signal, so that the type of image displayed by the display device 203 switches smoothly, preventing the surgeon from feeling uncomfortable and reducing the burden on the surgeon.

[0246] (Other embodiments) Furthermore, in the first to third embodiments of the present disclosure, a treatment for turbidity due to bone powder or the like in a liquid such as a perfusion solution has been described, but the present disclosure is not limited to a liquid, and can be applied to an air environment as well. The first to third embodiments can also be applied to a reduction in visibility in the field of view of an endoscope due to cutting debris, fat mist, etc., generated during an air-borne treatment at a joint site.

[0247] Furthermore, in the first to third embodiments of the present disclosure, treatments on the knee joint have been described, but the present disclosure can also be applied to other parts (such as the spine) in addition to the knee joint.

[0248] Furthermore, the first to third embodiments of the present disclosure can be applied to turbidity other than bone powder, such as debris of soft tissue, synovial membrane, fat, etc., and other noise (cavitation such as bubbles). For example, the first to third embodiments can be applied to turbidity or visual field degradation caused by tissue fragments such as cartilage, synovial membrane, fat, etc., which are factors that cause visual field degradation due to treatment with the treatment tool 301.

[0249] Furthermore, the first to third embodiments of the present disclosure can also be applied to treatment in liquid using the treatment tool 301, which can reduce deterioration of the field of view due to fine bubbles generated by factors such as cavitation caused by ultrasonic vibrations of the treatment tool 301.

[0250] Furthermore, the first to third embodiments of the present disclosure can be applied even when the field of view of the endoscope 201 is obstructed by a relatively large piece of tissue. In this case, the endoscope control device 202 may determine whether the field of view of the endoscope 201 is obstructed by an obstruction based on the first image, and if it is determined that the field of view is obstructed by an obstruction, perform image processing to remove the obstruction using well-known technology. In this case, the endoscope control device 202 may perform image processing to an extent that does not affect processing, using the size of the treatment area by the treatment tool 301, the time the treatment target site 100 is obstructed, etc.

[0251] Furthermore, the first to third embodiments of the present disclosure can also be applied to cases where a filter that can transmit near-infrared light (700 nm to 2500 nm) or an LED that can irradiate near-infrared light is used instead of infrared light.

[0252] Furthermore, in the first to third embodiments of the present disclosure, the composite image generation unit 2228 may generate a composite image by combining the second corrected image and the first image, or may generate a composite image by combining the second corrected image and the first corrected image. Furthermore, various inventions can be realized by appropriately combining multiple components disclosed in the treatment systems according to the first to third embodiments of the present disclosure. For example, some components may be omitted from all the components described in the treatment systems according to the first to third embodiments of the present disclosure. Furthermore, the components described in the treatment systems according to the first to third embodiments of the present disclosure may be appropriately combined.

[0253] Furthermore, in the treatment systems according to the first to third embodiments of the present disclosure, the above-described "unit" can be read as "means" or "circuit," etc. For example, a control unit can be read as control means or a control circuit.

[0254] In addition, the programs to be executed by the treatment systems according to embodiments 1 to 3 of the present disclosure are provided as file data in an installable or executable format stored in a computer-readable storage medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

[0255] Furthermore, the programs executed by the treatment systems according to the first to third embodiments of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0256] In the description of the flowcharts in this specification, expressions such as "first," "then," and "continue" are used to clearly indicate the order of processing between steps, but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed within a consistent range. Furthermore, programs are not limited to those consisting of simple branching processing, and branching can be achieved by comprehensively determining more judgment items.

[0257] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure of the present invention. [Explanation of symbols]

[0258] 1 Treatment System 2 Endoscopic devices 3 Treatment Device 4 Guiding Devices 5 Perfusion device 6,7 Lighting equipment 7 Network Control Device 8 Network Server 201, 201A, 201B Endoscope 202 Endoscope control device 203 Display device 204,2244 Imaging unit 205 Operation input section 211 Insertion part 221 Imaging processing unit 222 Image Processing Unit 223 Turbidity detection unit 224a Image sensor 227,315,326,606 CPU 228,316,327,607,2231 memory 301 Treatment tools 302 Treatment tool control device 303 Foot Switch 311 Treatment tool body 312 Ultrasound Probe 312a Ultrasonic vibrator 401 Guide body 601 First lighting control unit 602 Second lighting control unit 603 First lighting device 604 Second lighting device 800 Lighting Department 801 Light source 802 Rotating Filter 802a Red filter 802b Green filter 802c Blue Filter 802d IR filter 2221 Image data input unit 2222 First image generation unit 2223 First detection unit 2224 Second Image Generation Unit 2225 Second detection unit 2226 First corrected image generation unit 2226a Turbidity estimation section 2226b Local histogram generation unit 2226c Statistical information calculation section 2226d Correction coefficient calculation unit 2226e Contrast correction section 2227 Second corrected image generation unit 2228 Synthetic Image Generation Unit 2229 Display image generation unit 2230 Turbidity determination unit 2232 Image processing control unit 2241 Image sensor 2241a Pixel section 2241b Color filter 2242 First Imaging Unit 2243 Second Imaging Unit

Claims

1. a first image acquisition unit that acquires first image data that includes at least a region of the living body to be treated by the energy treatment device; a first detector that detects a change in gradation from at least a partial region of a first image corresponding to the first image data; a first corrected image generating unit that generates first corrected image data by performing gradation correction on the first image based on a detection result of the first detecting unit; a display image generating unit that generates a display image based on the first corrected image data; Equipped with The first detection unit detecting turbidity contained in the first image as a change in gradation; Image processing device.

2. 2. The image processing device according to claim 1, a second image acquisition unit that acquires second image data that is different from the wavelength band of the first image and includes a part of the region; a second detection unit that detects a change in gradation from at least a partial region of a second image corresponding to the second image data; a second corrected image generating unit that generates second corrected image data by performing gradation correction on the second image based on a detection result of the second detecting unit; Furthermore, The display image generation unit generating a display image based on at least one of the first corrected image data and the second corrected image data; Image processing device.

3. 3. The image processing device according to claim 2, a composite image generating unit that generates composite image data by combining the first corrected image data and the second corrected image data, The display image generation unit generating a display image based on the composite image data; Image processing device.

4. 3. The image processing device according to claim 2, The display image generation unit generating a first display image based on the first corrected image data and a second display image based on the second corrected image data; outputting both or either one of the first display image and the second display image to a display device; Image processing device.

5. 2. The image processing device according to claim 1, The display image generation unit generating the display image based on a synchronization signal synchronized with imaging driving of an imaging device; Image processing device.

6. 2. The image processing device according to claim 1, The first corrected image generation unit generating the first corrected image data by performing gradation correction on the first image based on a drive signal of an energy treatment tool that treats the living body and a detection result of the first detection unit; Image processing device.

7. 2. The image processing device according to claim 1, a turbidity determination unit that determines whether the turbidity detected by the first detection unit is equal to or greater than a predetermined value, The display image generation unit When the turbidity determination unit determines that the turbidity is equal to or greater than a predetermined value, a display image is generated based on the first corrected image data, generating a display image based on the first image data when the turbidity determination unit determines that the turbidity is not equal to or greater than a predetermined value; Image processing device.

8. 2. The image processing device according to claim 1, The first corrected image generation unit a turbidity estimation unit that estimates a turbidity component for each pixel in the first image; a histogram generating unit that generates a brightness histogram based on the first image and the turbidity component estimated by the turbidity estimating unit; a representative luminance calculation unit that calculates a representative luminance based on the luminance histogram generated by the histogram generation unit; a correction coefficient calculation unit that calculates a correction coefficient for correcting the contrast of the first image based on the turbidity component estimated by the turbidity estimation unit and the representative luminance calculated by the representative luminance calculation unit; a contrast correction unit that performs contrast correction of a reference pixel on the first image based on the turbidity component and the correction coefficient to generate the first corrected image data; having Image processing device.

9. 3. The image processing device according to claim 2, The second image acquisition unit acquiring, as the second image data, image data generated by an image sensor capable of receiving invisible light including at least an infrared wavelength band; Image processing device.

10. 10. The image processing device according to claim 9, The second corrected image generation unit generating the second corrected image data by performing edge enhancement processing on the second image; Image processing device.

11. The image processing device according to claim 10, The first corrected image generation unit a turbidity estimation unit that estimates a turbidity component for each pixel in the first image; a histogram generating unit that generates a brightness histogram based on the first image and the turbidity component estimated by the turbidity estimating unit; a representative luminance calculation unit that calculates a representative luminance based on the luminance histogram generated by the histogram generation unit; a correction coefficient calculation unit that calculates a correction coefficient for correcting the contrast of the first image based on the turbidity component estimated by the turbidity estimation unit and the representative luminance calculated by the representative luminance calculation unit; a contrast correction unit that performs contrast correction of a reference pixel on the first image based on the turbidity component and the correction coefficient to generate the first corrected image data; having Image processing device.

12. The image processing device according to claim 11, a composite image generating unit that generates composite image data by combining the first corrected image data and the second corrected image data, The display image generation unit generating a display image based on the composite image data; Image processing device.

13. 2. The image processing device according to claim 1, The first image acquisition unit acquiring, as the first image data, image data generated by an image sensor capable of receiving visible light in a visible wavelength band and invisible light in a wavelength band outside the visible range; Image processing device.

14. 3. The image processing device according to claim 2, The first image acquisition unit acquiring, as the first image data, image data generated by a first image sensor capable of receiving visible light in a visible wavelength band; The second image acquisition unit acquiring, as the second image data, image data generated by a second image sensor capable of receiving invisible light in a wavelength band outside the visible range; Image processing device.

15. 2. The image processing device according to claim 1, a second image acquisition unit that acquires second image data having a wavelength different from that of the first image; a second detection unit that detects a change in gradation from at least a partial region of a second image corresponding to the second image data; a second corrected image generating unit that generates second corrected image data by performing gradation correction on the second image based on a detection result of the second detecting unit; Furthermore, The display image generation unit generating a display image based on at least one of the first corrected image data and the second corrected image data; Image processing device.

16. a first image acquisition unit that acquires a first image corresponding to first image data including a region of a living body to be treated by the energy treatment device; a second image acquisition unit that acquires a second image corresponding to second image data having a wavelength different from that of the first image; a first detector that detects a change in gradation from at least a partial region of the first image; a corrected image generating unit that generates corrected image data by performing gradation correction on the second image based on a detection result of the first detecting unit; a display image generation unit that generates a display image based on the corrected image data; Equipped with The first detection unit detecting turbidity contained in the first image as a change in gradation; Image processing device.

17. an energy treatment device that can be inserted into a subject and can treat a treatment target site; an endoscope that can be inserted into the subject and that can capture an image of at least the treatment target site to generate first image data; an image processing device that performs image processing on the first image data and outputs the image data to a display device; Equipped with The image processing device includes: a first image acquisition unit that acquires the first image data; a first detector that detects a change in gradation from at least a partial region of a first image corresponding to the first image data; a first corrected image generating unit that generates first corrected image data by performing gradation correction on the first image based on a detection result of the first detecting unit; a display image generating unit that generates a display image based on the first corrected image data; Equipped with The first detection unit detecting turbidity contained in the first image as a change in gradation; Treatment system.

18. An image processing method executed by an image processing device equipped with a processor having hardware, the processor: acquiring first image data including a region of the living body to be treated by the energy treatment device; detecting a change in gradation from at least a partial region of a first image corresponding to the first image data, and detecting turbidity contained in the first image as a change in gradation; generating first corrected image data by correcting the gradation of the first image based on a detection result of detecting a change in gradation from at least a partial region of the first image; generating a display image based on the first corrected image data; To carry out the Image processing methods.

Images