Image processing device, processing system and image processing method
Patent Information
- Application Number
- JP2024505781
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-03-10
- Filing Date
- 2022-03-10
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2042-03-10
AI Technical Summary
In arthroscopic surgery, the generation of bone powder from ultrasonic treatment instruments clouds the field of view of the endoscope, necessitating treatment pauses and prolonging surgical time.
An image processing device enhances image data to maintain a clear field of view by detecting and emphasizing the energy treatment tool area, generating enhanced images for display, and integrating an indicator portion on the energy treatment device to improve visibility.
Continues treatment despite deteriorated endoscope visibility, ensuring uninterrupted surgical procedures.
Description
[Technical Field]
[0001] The present disclosure relates to an image processing device, an energy treatment device, a treatment system, and an image processing method. [Background technology]
[0002] In arthroscopic surgery, a technique is known in which the inside of the joint is inflated with an irrigating fluid such as saline using an irrigation device to ensure a field of view and then the treatment area is treated (see, for example, Patent Document 1). In this technique, the hammering action of an ultrasonic treatment instrument breaks down the bone, generating bone powder (bone shavings) and cerebrospinal fluid, so the irrigating fluid is used to send the bone powder and cerebrospinal fluid out of the field of view of the endoscope to ensure a field of view of the treatment area. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 4564595 Summary of the Invention [Problem to be solved by the invention]
[0004] In arthroscopic surgery, when bones are continuously crushed by the hammering action of an ultrasonic treatment instrument, a large amount of bone powder is generated. This bone powder disperses in the irrigation fluid, making the fluid cloudy and obstructing the field of view of the arthroscope used to observe the treatment area, making it difficult to see the treatment area.
[0005] However, in the above-mentioned Patent Document 1, if the field of view of the endoscope observing the treatment area becomes cloudy and deteriorates, the perfusion fluid will expel bone powder from the field of view of the endoscope, and treatment on the treatment area must be stopped and the surgeon must wait until the field of view of the endoscope improves, which extends the treatment time and places a burden on both the surgeon and the patient.
[0006] The present disclosure has been made in consideration of the above, and aims to provide an image processing device, an energy treatment device, a treatment system, and an image processing method that enable treatment of a treatment area to be continued even when the field of view of the endoscope deteriorates. [Means for solving the problem]
[0007] In order to solve the above-mentioned problems and achieve the object, the image processing device according to the present disclosure includes an image acquisition unit that acquires image data including at least a portion of an area where a living body is to be treated by an energy treatment tool, a detection unit that detects an area including at least a portion of the energy treatment tool from an image corresponding to the image data, an enhancement image generation unit that generates enhancement image data in which the area is enhanced compared to other areas based on the image data and the detection result detected by the detection unit, and a display image generation unit that generates a display image based on the enhancement image data.
[0008] Moreover, the energy treatment device according to the present disclosure is an energy treatment device that can be inserted into a subject, can treat a treatment target site, and is imaged by an endoscope inserted into the subject, and is provided with an indicator portion at the tip that can be detected by emphasizing edge components and brightness of a region that is included in a part of the energy treatment device compared to other regions in an image corresponding to image data imaged by the endoscope.
[0009] Furthermore, a treatment system according to the present disclosure includes an energy treatment device that can be inserted into a subject and can treat a treatment target site, an endoscope that can be inserted into the subject and can capture an image of at least an area of the living body to be treated with the energy treatment device and generate image data, and an image processing device that performs image processing on the image data and outputs the image data to a display device, wherein the image processing device includes an image acquisition unit that acquires the image data, a detection unit that detects an area including at least a part of the energy treatment device from an image corresponding to the image data, an emphasis image generation unit that generates emphasis image data in which the area is emphasized compared to other areas based on the image data and the detection result detected by the detection unit, and a display image generation unit that generates a display image based on the emphasis image data.
[0010] Furthermore, the image processing method according to the present disclosure is an image processing method executed by an image processing device provided with a processor having hardware, in which the processor acquires image data including at least a portion of an area to be treated on a living body by an energy treatment tool, detects an area including at least a portion of the energy treatment tool from an image corresponding to the image data, generates enhanced image data that enhances the area compared to other areas based on the image data and the detection result of the area, and generates a display image based on the enhanced image data. [Effects of the Invention]
[0011] According to the present disclosure, it is possible to achieve the effect of continuing treatment on the treatment site even when the field of view of the endoscope deteriorates. [Brief explanation of the drawings]
[0012] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a treatment system according to a first embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a state in which a bone hole is formed by the ultrasonic probe according to the first embodiment of the present disclosure. [Figure 3A]FIG. 3A is a schematic diagram illustrating a schematic configuration of an ultrasound probe according to the first embodiment of the present disclosure. [Figure 3B] FIG. 3B is a schematic view of the direction of arrow A in FIG. 3A. [Figure 4] FIG. 4 is a block diagram illustrating an outline of the functional configuration of the entire treatment system according to the first embodiment of the present disclosure. [Figure 5] FIG. 5 is a block diagram illustrating a detailed functional configuration of the endoscope apparatus according to the first embodiment of the present disclosure. [Figure 6A] FIG. 6A is a diagram showing a state in which the field of view of the endoscope according to the first embodiment of the present disclosure is good. [Figure 6B] FIG. 6B is a diagram illustrating a state in which the field of view of the endoscope according to the first embodiment of the present disclosure is poor. [Figure 7] FIG. 7 is a block diagram illustrating a detailed functional configuration of the processing device according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram illustrating a detailed functional configuration of the perfusion device according to the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a block diagram illustrating a detailed functional configuration of the illumination device according to the first embodiment of the present disclosure. [Figure 10] FIG. 10 is a block diagram illustrating a detailed functional configuration of the image processing unit according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram illustrating a detailed functional configuration of the corrected image generation unit according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a flowchart outlining a procedure performed by an operator using the treatment system according to the first embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating an overview of a process executed by the endoscope control device in a cutting procedure according to the first embodiment of the present disclosure. [Figure 14] FIG. 14 is a flowchart showing a detailed outline of the turbidity response control process of FIG. [Figure 15] FIG. 15 is a diagram illustrating an example of a display image when the field of view of the endoscope is good according to the first embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram showing the relationship between the position on the line AA' in FIG. 15 and the luminance. [Figure 17] FIG. 17 is a diagram illustrating an example of a display image when the field of view of the endoscope is poor according to the first embodiment of the present disclosure. [Figure 18] FIG. 18 is a diagram showing the relationship between the position on the line AA' in FIG. 16 and the luminance. [Figure 19] FIG. 19 is a diagram showing the relationship between the position on the line AA' and the luminance after the corrected image generation unit has performed gradation correction on the display image of FIG. 17 according to the first embodiment of the present disclosure. [Figure 20] FIG. 20 is a diagram illustrating the relationship between the position and brightness of an enhanced image generated by an enhanced image generating unit according to the first embodiment of the present disclosure. [Figure 21] FIG. 21 is a diagram schematically illustrating a method for generating a composite image generated by a composite image generation unit according to the first embodiment of the present disclosure. [Figure 22] FIG. 22 is a diagram illustrating an example of a display image displayed by the display device in the first embodiment of the present disclosure. [Figure 23] FIG. 23 is a block diagram illustrating a functional configuration of an image processing unit according to the second embodiment of the present disclosure. [Figure 24] FIG. 24 is a flowchart illustrating a detailed outline of the turbidity response control process executed by the endoscope control device according to the second embodiment of the present disclosure. [Figure 25] FIG. 25 is a diagram schematically illustrating a method for generating a composite image generated by a composite image generating unit according to the second embodiment of the present disclosure. [Figure 26] FIG. 26 is a block diagram illustrating a functional configuration of an image processing unit according to the third embodiment of the present disclosure. [Figure 27] FIG. 27 is a schematic diagram showing a schematic configuration of a part of a treatment tool according to the third embodiment of the present disclosure. [Figure 28] FIG. 28 is a diagram illustrating an example of a display image in which a part of a treatment tool is displayed in the field of view of an endoscope according to the third embodiment of the present disclosure. [Figure 29] FIG. 29 is a diagram showing the relationship between the position on the line AA' in FIG. 28 and the luminance. [Figure 30] FIG. 30 is a diagram illustrating the relationship between the position on the same line as the line AA' in FIG. 28 and the luminance in an HDR image generated by an HDR image generation unit according to Embodiment 3 of the present disclosure. [Figure 31] FIG. 31 is a diagram illustrating the relationship between the position on the same line as the line AA' in FIG. 28 and the brightness in a displayed image when the field of view of the endoscope according to the third embodiment of the present disclosure is poor. [Figure 32] FIG. 32 is a diagram illustrating the relationship between the position on the same line as the line AA' in FIG. 28 and the luminance in a composite image generated by a composite image generating unit according to Embodiment 3 of the present disclosure. [Figure 33] FIG. 33 is a block diagram illustrating a functional configuration of an image processing device according to the fourth embodiment of the present disclosure. [Figure 34] FIG. 34 is a block diagram showing a functional configuration of a treatment tool according to the fourth embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION
[0013] Hereinafter, embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. Furthermore, each drawing referred to in the following description merely shows a schematic representation of the shape, size, and positional relationship to the extent that the contents of the present disclosure can be understood. In other words, the present disclosure is not limited to only the shape, size, and positional relationship exemplified in each drawing. Furthermore, in the following description, the same parts in the drawings will be denoted by the same reference numerals.
[0014] (Embodiment 1) [Outline of processing system] FIG. 1 is a diagram showing a schematic configuration of a treatment system 1 according to the first embodiment. The treatment system 1 shown in Fig. 1 applies ultrasonic vibrations to biological tissue such as bone to treat the biological tissue. Here, treatment refers to, for example, removal or cutting of biological tissue such as bone. Note that Fig. 1 illustrates a treatment system for performing anterior cruciate ligament reconstruction as the treatment system 1.
[0015] The treatment system 1 shown in FIG. 1 includes an endoscope apparatus 2, a treatment apparatus 3, a guiding device 4, a perfusion apparatus 5, and an illumination apparatus 6.
[0016] [Configuration of the endoscope device] First, the configuration of the endoscope device 2 will be described. The endoscope device 2 includes an endoscope 201 , an endoscope control device 202 , and a display device 203 .
[0017] The endoscope 201 has a distal end portion of an insertion section 211 inserted into the joint cavity C1 through a first portal P1 that communicates between the inside of the joint cavity C1 of the knee joint J1 of the subject and the outside of the skin. The endoscope 201 illuminates the inside of the joint cavity C1, takes in illumination light (subject image) reflected within the joint cavity C1, and captures the subject image to generate image data.
[0018] The endoscope control device 202 performs various image processing on image data captured by the endoscope 201, and displays a display image corresponding to the image data after this image processing on the display device 203. The endoscope control device 202 is connected to the endoscope 201 and the display device 203 by wire or wirelessly.
[0019] The display device 203 receives data, image data (display image), audio data, etc. transmitted from each device constituting the treatment system 1 via the endoscope control device 202, and displays, notifies, and outputs a display image according to the received data. The display device 203 is configured using a display panel made of liquid crystal or organic EL (Electro-Luminescence).
[0020] [Configuration of processing device] Next, the configuration of the processing device 3 will be described. The treatment device 3 includes a treatment tool 301 , a treatment tool control device 302 , and a foot switch 303 .
[0021] The treatment tool 301 includes a treatment tool body 311, an ultrasonic probe 312 (see FIG. 2, which will be described later), and a sheath 313.
[0022] The treatment tool body 311 is formed in a cylindrical shape. The treatment tool body 311 also houses an ultrasonic transducer 312a (see FIG. 2, which will be described later) that is made up of a bolt-clamped Langevin-type transducer and generates ultrasonic vibrations in response to supplied driving power.
[0023] The treatment tool control device 302 supplies driving power to the ultrasonic transducer 312a in response to an operator's operation of the foot switch 303. The supply of driving power is not limited to the operation of the foot switch 303, and may be performed in response to, for example, an operation of an operation unit (not shown) provided on the treatment tool 301.
[0024] The foot switch 303 is an input interface that is operated by the operator with his / her foot when driving the ultrasonic probe 312 .
[0025] Next, the ultrasonic probe 312 will be described. FIG. 2 is a diagram showing how a bone hole 101 is formed by an ultrasonic probe 312. As shown in FIG. FIG. 3A is a schematic diagram showing the general configuration of the ultrasonic probe 312. As shown in FIG. FIG. 3B is a schematic view of the direction of arrow A in FIG. 3A. In the first embodiment, the ultrasonic probe 312 functions as an energy treatment tool.
[0026] As shown in FIGS. 2, 3A, and 3B, the ultrasonic probe 312 is made of, for example, a titanium alloy and has a substantially cylindrical shape. The proximal end of the ultrasonic probe 312 is connected to an ultrasonic vibrator 312a inside the treatment instrument main body 311. The ultrasonic probe 312 transmits ultrasonic vibrations generated by the ultrasonic vibrator 312a from the proximal end to the distal end. Specifically, the ultrasonic vibrations in the first embodiment are longitudinal vibrations along the longitudinal direction of the ultrasonic probe 312 (the vertical direction in FIG. 2). The distal end of the ultrasonic probe 312 is provided with the ultrasonic vibrator 312a, as shown in FIG. 2.
[0027] The sheath 313 is formed in a cylindrical shape that is longer and thinner than the treatment instrument body 311, and covers a part of the outer periphery of the ultrasonic probe 312 from the treatment instrument body 311 to an arbitrary length.
[0028] The ultrasonic vibrator 312a of the ultrasonic probe 312 in the treatment instrument 301 configured in this manner is inserted into the joint cavity C1 while being guided by the guiding device 4 inserted into the joint cavity C1 through a second portal P2 that connects the inside of the joint cavity C1 with the outside of the skin.
[0029] Next, the treatment tool 301 generates ultrasonic vibrations while the ultrasonic vibrator 312a of the ultrasonic probe 312 is in contact with the bone treatment target area 100, and the hammering action causes the part of the bone that mechanically collides with the ultrasonic vibrator 312a to be crushed into fine particles (see Figure 2).
[0030] Thereafter, when the operator presses the ultrasonic vibrator 312a of the ultrasonic probe 312 into the treatment target portion 100, the ultrasonic vibrator 312a advances into the treatment target portion 100 while crushing the bone. As a result, a bone hole 101 is formed in the treatment target portion 100.
[0031] Further, at the proximal end of the treatment tool body 311, a circuit board 317 on which a posture detection section 314, a CPU (Central Processing Unit) 315, and a memory 316 are mounted is provided (see FIGS. 3A and 3B).
[0032] The posture detection unit 314 includes a sensor that detects the rotation and movement of the treatment tool 301. The posture detection unit 314 detects three mutually orthogonal axes, including an axis parallel to the longitudinal axis of the ultrasonic probe 312, and the rotation around each axis. The treatment tool control device 302 described above determines that the treatment tool 301 is stationary if the detection result of the posture detection unit 314 does not change for a certain period of time.
[0033] The CPU 315 controls the operation of the attitude detection unit 314 and transmits and receives information to and from the treatment tool control device 302. The CPU 315 reads a program stored in the memory 316 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby enabling the hardware and software to work together and realizing a functional module that meets a predetermined purpose.
[0034] [Configuration of the guiding device] Next, the configuration of the guiding device 4 will be described. The guiding device 4 is inserted into the joint cavity C1 through the second portal P2, and guides the insertion of the tip portion of the ultrasonic probe 312 of the treatment tool 301 into the joint cavity C1.
[0035] The guiding device 4 includes a guide body 401, a handle portion 402, and a drainage portion 403 with a cock.
[0036] The guide body 401 has a cylindrical shape and has a through-hole 401a therein through which the ultrasonic probe 312 is inserted (see FIG. 1). The guide body 401 restricts the advancement of the ultrasonic probe 312 inserted into the through-hole 401a in a fixed direction and guides the movement of the ultrasonic probe 312. In the first embodiment, the cross-sectional shapes of the outer and inner peripheral surfaces of the guide body 401 perpendicular to the central axis are each approximately circular. Furthermore, the guide body 401 becomes thinner toward the tip. In other words, the tip surface 401b of the guide body 401 is an inclined surface that intersects obliquely with the central axis.
[0037] The cock-equipped drainage section 403 is provided on the outer peripheral surface of the guide main body 401 and has a cylindrical shape that communicates with the inside of the guide main body 401. One end of a drainage tube 505 of the perfusion device 5 is connected to the cock-equipped drainage section 403, forming a flow path that communicates between the guide main body 401 and the drainage tube 505 of the perfusion device 5. This flow path is configured to be openable and closable by operating a cock (not shown) provided in the cock-equipped drainage section 403.
[0038] [Configuration of the perfusion device] Next, the configuration of the perfusion device 5 will be described. The perfusion device 5 delivers a perfusion fluid such as sterilized physiological saline into the joint cavity C1 and also discharges the perfusion fluid to the outside of the joint cavity C1.
[0039] The perfusion device 5 includes a liquid source 501, a liquid supply tube 502, a liquid supply pump 503, a drain bottle 504, a drain tube 505, and a drain pump 506 (see FIG. 1).
[0040] Fluid source 501 contains an irrigation fluid therein. Fluid source 501 is connected to fluid supply tube 502. The irrigation fluid is sterilized physiological saline or the like. Fluid source 501 is configured using, for example, a bottle or the like.
[0041] The liquid supply tube 502 has one end connected to the liquid source 501 and the other end connected to the endoscope 201 .
[0042] The fluid supply pump 503 supplies the perfusion fluid from the fluid source 501 to the endoscope 201 through the fluid supply tube 502. The perfusion fluid supplied to the endoscope 201 is supplied into the joint cavity C1 from a fluid supply hole formed at the tip of the insertion section 211.
[0043] The drainage bottle 504 stores the irrigation fluid discharged to the outside of the joint cavity C1. A drainage tube 505 is connected to the drainage bottle 504.
[0044] The drainage tube 505 has one end connected to the guiding device 4 and the other end connected to the drainage bottle 504 .
[0045] The drainage pump 506 drains the perfusion fluid in the joint cavity C1 through a flow path from the guiding device 4 inserted into the joint cavity C1 to the drainage tube 505 into the drainage bottle 504. Although the first embodiment is described using the drainage pump 506, the present invention is not limited to this, and a suction device provided in the facility may also be used.
[0046] [Configuration of lighting device] Next, the configuration of the lighting device 6 will be described. The illumination device 6 has two light sources that emit two illumination lights with different wavelength bands. The two illumination lights are, for example, visible white light and invisible infrared or special light. Information based on wavelengths different from visible light can be obtained from the invisible light illumination. For example, in a murky imaging environment, an infrared image captured under infrared illumination can have higher contrast than a normal image captured under white light illumination. Therefore, by extracting edge components from the image captured under infrared illumination and superimposing the extracted edge components on a normal image captured under white light illumination or a corrected image obtained by correcting the normal image, an image with further enhanced edges can be generated. The illumination light from the illumination device 6 is propagated to the endoscope 201 via a light guide and emitted from the tip of the endoscope 201.
[0047] [Functional configuration of the entire treatment system] Next, the functional configuration of the entire treatment system will be described. FIG. 4 is a block diagram showing an outline of the functional configuration of the entire treatment system 1. As shown in FIG. In addition to the configuration described above (see FIG. 1), the treatment system 1 shown in FIG. 4 further includes a network control device 7 that controls communication throughout the system, and a network server 8 that stores various data.
[0048] The network control device 7 is communicably connected to the endoscope device 2, treatment device 3, perfusion device 5, illumination device 6, and network server 8. While Fig. 4 illustrates a case where the devices are connected wirelessly, they may also be connected by wire. The detailed functional configurations of the endoscope device 2, treatment device 3, perfusion device 5, and illumination device 6 will be described below.
[0049] The network server 8 is communicably connected to the endoscope device 2, the treatment device 3, the perfusion device 5, the illumination device 6, and the network control device 7. The network server 8 stores various data of each device constituting the treatment system 1. The network server 8 is configured using a processor having hardware such as a CPU, and memories such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).
[0050] [Functional configuration of the endoscope device] Next, the functional configuration of the above-mentioned endoscope device 2 will be described. FIG. 5 is a block diagram showing a detailed functional configuration of the endoscope device 2. As shown in FIG. As shown in FIGS. 4 and 5, the endoscope device 2 includes an endoscope control device 202, a display device 203, an imaging unit 204 provided in the endoscope 201, and an operation input unit 205.
[0051] The endoscope control device 202 includes an imaging processing unit 221, an image processing unit 222, a turbidity detection unit 223, an input unit 226, a CPU 227, a memory 228, a wireless communication unit 229, a distance sensor driving circuit 230, a distance data memory 231, and a communication interface 232.
[0052] The imaging processing unit 221 has an imaging element drive control circuit 221a that controls the drive of an imaging element 2241 included in the imaging unit 204 provided in the endoscope 201, and an imaging element signal control circuit 221b that is provided in a patient circuit 202b electrically insulated from the primary circuit 202a and controls the signal of the imaging element 2241. The imaging element drive control circuit 221a is provided in the primary circuit 202a. The imaging element signal control circuit 221b is provided in a patient circuit 202b electrically insulated from the primary circuit 202a.
[0053] The image processing unit 222 performs predetermined image processing on image data (RAW data) input via a bus and outputs the result to the display device 203. The image processing unit 222 is configured using a processor having hardware such as a DSP (Digital Signal Processor) or FPGA (Field-Programmable Gate Array). The image processing unit 222 reads a program stored in a memory 228 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby enabling the hardware and software to work together and realize a functional module that meets a predetermined purpose. The detailed functional configuration of the image processing unit 222 will be described later.
[0054] The turbidity detection unit 223 detects turbidity of the field of view of the endoscope 201 in the joint cavity C1 based on information about turbidity of the field of view of the endoscope 201. Here, the information about turbidity is, for example, a value obtained from image data generated by the endoscope 201, a physical property value (turbidity) of the perfusion fluid, impedance obtained from the treatment device 3, etc.
[0055] FIG. 6A is a diagram showing a state in which the field of view of the endoscope 201 is good. FIG. 6B is a diagram showing a state in which the field of view of the endoscope 201 is poor. 6A and 6B are diagrams each showing a display image corresponding to image data representing the field of view of the endoscope 201 when the surgeon forms a bone hole in the femoral lateral condyle 900. Of these, Fig. 6B shows a state in which the field of view of the endoscope 201 is blurred due to bone being crushed into fine particles by driving the ultrasonic probe 312. In Fig. 6B, the fine bones are represented by dots.
[0056] The input unit 226 receives the input of a signal input by the operation input unit 205 and the input of a signal from each device constituting the treatment system 1 .
[0057] The CPU 227 comprehensively controls the operation of the endoscope control device 202. The CPU 227 reads out a program stored in the memory 228 into a working area of the memory and executes it, and controls each component part through the execution of the program by the processor, thereby controlling the operation of each part of the endoscope control device 202 through cooperation between hardware and software.
[0058] The memory 228 stores various information necessary for the operation of the endoscope control device 202, various programs executed by the endoscope control device 202, image data captured by the imaging unit 204, etc. The memory 228 is configured using, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a frame memory, etc.
[0059] The wireless communication unit 229 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi (registered trademark) or Bluetooth (registered trademark).
[0060] The distance sensor driving circuit 230 drives a distance sensor (not shown) that measures the distance to a predetermined object in an image captured by the imaging unit 204. In the first embodiment, the distance sensor may be provided in the imaging element 2241. In this case, the imaging element 2241 may be provided with phase difference pixels capable of measuring the distance from the imaging element 2241 to the predetermined object, instead of effective pixels. Of course, a ToF (Time of Flight) sensor or the like may be provided near the tip of the endoscope 201.
[0061] Distance data detected by the distance sensor is stored in the distance data memory 231. The distance data memory 231 is configured using, for example, RAM and ROM.
[0062] The communication interface 232 is an interface for communicating with the image capturing unit 204 .
[0063] Of the above-described components, all except the imaging element signal control circuit 221b are provided in the primary circuit 202a and are interconnected by bus wiring.
[0064] The imaging unit 204 is provided in the endoscope 201. The imaging unit 204 has an imaging element 2241, a CPU 242, and a memory 243.
[0065] Under the control of the CPU 242, the image sensor 2241 generates image data by capturing an image of a subject formed by one or more optical systems (not shown), and outputs the generated image data to the endoscope control device 202. The image sensor 2241 is configured using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor.
[0066] The CPU 242 comprehensively controls the operation of the imaging unit 204. The CPU 242 reads out a program stored in the memory 243 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby controlling the operation of the imaging unit 204 through cooperation between hardware and software.
[0067] The memory 243 stores various information necessary for the operation of the imaging unit 204, various programs executed by the endoscope 201, image data generated by the imaging unit 204, etc. The memory 243 is configured using RAM, ROM, frame memory, etc.
[0068] The operation input unit 205 is configured using input interfaces such as a mouse, keyboard, touch panel, and microphone, and receives operation inputs of the endoscope device 2 from the operator.
[0069] [Functional configuration of the processing device] Next, the functional configuration of the processing device 3 will be described. FIG. 7 is a block diagram showing a detailed functional configuration of the processing device 3. As shown in FIG. As shown in FIGS. 4 and 7, the treatment device 3 includes a treatment tool 301, a treatment tool control device 302, and an input / output unit 304.
[0070] The treatment tool 301 includes an ultrasonic transducer 312 a , a posture detection unit 314 , a CPU 315 , and a memory 316 .
[0071] The attitude detection unit 314 detects the attitude of the treatment tool 301 and outputs the detection result to the CPU 315. The attitude detection unit 314 is configured using at least one of an acceleration sensor and an angular velocity sensor.
[0072] The CPU 315 comprehensively controls the operation of the treatment tool 301 including the ultrasonic transducer 312a. The CPU 315 reads out a program stored in the memory 316 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby allowing the hardware and software to work together and realizing a functional module that meets a predetermined purpose.
[0073] The memory 316 stores various information necessary for the operation of the treatment tool 301, various programs executed by the treatment tool 301, and identification information for identifying the type, manufacturing date, performance, etc. of the treatment tool 301.
[0074] The treatment tool control device 302 includes a primary circuit 321, a patient circuit 322, a transformer 323, a first power supply 324, a second power supply 325, a CPU 326, a memory 327, a wireless communication unit 328, a communication interface 329, and an impedance detection unit 330.
[0075] The primary circuit 321 generates power to be supplied to the treatment tool 301 . The patient circuit 322 is electrically isolated from the primary circuit 321 . Transformer 323 electromagnetically connects primary circuit 321 and patient circuit 322 together. The first power supply 324 is a high-voltage power supply that supplies driving power to the treatment tool 301 . The second power supply 325 is a low-voltage power supply that supplies driving power to the control circuit in the treatment tool control device 302 .
[0076] The CPU 326 comprehensively controls the operation of the treatment tool control device 302. The CPU 326 reads out a program stored in the memory 327 into a working area of the memory and executes it, and controls each component through the execution of the program by the processor, thereby controlling the operation of each part of the treatment tool control device 302 through cooperation between hardware and software.
[0077] The memory 327 stores various information necessary for the operation of the treatment tool control device 302, various programs executed by the treatment tool control device 302, etc. The memory 327 is configured using RAM, ROM, etc.
[0078] The wireless communication unit 328 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi (registered trademark) and Bluetooth (registered trademark).
[0079] The communication interface 329 is an interface for communicating with the treatment tool 301 .
[0080] The impedance detection unit 330 detects the impedance when the treatment tool 301 is driven, and outputs the detection result to the CPU 326. Specifically, the impedance detection unit 330 is electrically connected, for example, between the first power source 324 and the primary circuit 321, and detects the impedance of the treatment tool 301 based on the frequency of the first power source 324 and outputs the detection result to the CPU 326.
[0081] The input / output unit 304 is configured using input interfaces such as a mouse, keyboard, touch panel, and microphone, and output interfaces such as a monitor and speaker, and outputs operation inputs of the endoscopic device 2 by the surgeon and various information to be notified to the surgeon.
[0082] [Functional configuration of the perfusion device] Next, the functional configuration of the perfusion device 5 will be described. FIG. 8 is a block diagram showing the detailed functional configuration of the perfusion device 5. As shown in Figures 4 and 8, the perfusion device 5 includes a fluid supply pump 503, a drainage pump 506, a fluid supply control unit 507, a drainage control unit 508, an input unit 509, a CPU 510, a memory 511, a wireless communication unit 512, a communication interface 513, a pump CPU 514, a pump memory 515, and a turbidity detection unit 516.
[0083] The liquid supply control unit 507 includes a first drive control unit 571 , a first drive power generation unit 572 , a first transformer 573 , and a liquid supply pump drive circuit 574 .
[0084] The first drive control unit 571 controls the driving of the first drive power generation unit 572 and the liquid feed pump drive circuit 574 .
[0085] The first drive power generation unit 572 generates drive power for the liquid feed pump 503 and supplies this drive power to the first transformer 573 .
[0086] The first transformer 573 electromagnetically connects the first drive power generation unit 572 and the liquid feed pump drive circuit 574 together.
[0087] In fluid supply control unit 507 configured in this manner, first drive control unit 571, first drive power generation unit 572, and first transformer 573 are provided in primary circuit 5a. In addition, fluid supply pump drive circuit 574 is provided in patient circuit 5b that is electrically insulated from primary circuit 5a.
[0088] The drainage control unit 508 includes a second drive control unit 581, a second drive power generation unit 582, a second transformer 583, and a drainage pump drive circuit 584.
[0089] The second drive control unit 581 controls the driving of the second drive power generation unit 582 and the drainage pump drive circuit 584 .
[0090] The second driving power generating unit 582 generates driving power for the drainage pump 506 and supplies the generated driving power to the second transformer 583 .
[0091] The second transformer 583 electromagnetically connects the second drive power generating unit 582 and the drainage pump drive circuit 584 together.
[0092] In drainage control unit 508 configured in this manner, second drive control unit 581, second drive power generation unit 582, and second transformer 583 are provided in primary circuit 5a. Furthermore, drainage pump drive circuit 584 is provided in patient circuit 5b that is electrically insulated from primary circuit 5a.
[0093] The input unit 509 receives input of signals from each device constituting the treatment system 1, and outputs the received signals to the CPU 510 and the pump CPU 514.
[0094] The CPU 510 and the pump CPU 514 work together to comprehensively control the operation of the perfusion device 5. The CPU 510 reads a program stored in the memory 511 into a working area of the memory and executes it, and controls each component part through the execution of the program by the processor, thereby controlling the operation of each part of the perfusion device 5 through cooperation between hardware and software.
[0095] The memory 511 stores various information necessary for the operation of the perfusion device 5 and various programs executed by the perfusion device 5. The memory 511 is configured using RAM, ROM, and the like.
[0096] The wireless communication unit 512 is an interface for wireless communication with other devices, and is configured using a communication module capable of Wi-Fi, Bluetooth, or the like.
[0097] The communication interface 513 is an interface for communicating with the liquid feed pump 503 and the endoscope 201 .
[0098] The pump memory 515 stores various information necessary for the operation of the liquid feed pump 503 and the liquid discharge pump 506 and various programs executed by the liquid feed pump 503 and the liquid discharge pump 506 .
[0099] The turbidity detection unit 516 detects the turbidity of the perfusion fluid based on one or more of the physical properties, absorbance, impedance, and resistance of the perfusion fluid flowing in the drainage tube 505, and outputs the detection result to the CPU 510.
[0100] In the perfusion device 5 configured as above, an input unit 509, a CPU 510, a memory 511, a wireless communication unit 512, a communication interface 513, and a turbidity detection unit 516 are provided in the primary circuit 5a. Furthermore, an in-pump CPU 514 and an in-pump memory 515 are provided in the pump 5c. The in-pump CPU 514 and the in-pump memory 515 may be provided in the vicinity of the liquid feed pump 503 or in the vicinity of the drainage pump 506.
[0101] [Functional configuration of lighting device] Next, the functional configuration of the lighting device 6 will be described. FIG. 9 is a block diagram showing a detailed functional configuration of the lighting device 6. As shown in FIG. As shown in Figures 4 and 9, the lighting device 6 includes a first lighting control unit 601, a second lighting control unit 602, a first lighting device 603, a second lighting device 604, an input unit 605, a CPU 606, a memory 607, a wireless communication unit 608, a communication interface 609, a CPU 610 in the lighting circuit, and a memory 630 in the lighting circuit.
[0102] The first illumination control unit 601 has a first drive control unit 611 , a first drive power generation unit 612 , a first controller 613 , and a first drive circuit 614 .
[0103] The first drive control unit 611 controls the driving of the first drive power generation unit 612 , the first controller 613 and the first drive circuit 614 .
[0104] The first drive power generation section 612 generates drive power for the first lighting device 603 under the control of the first drive control section 611 , and outputs this drive power to the first controller 613 .
[0105] The first controller 613 controls the first drive circuit 614 in accordance with the drive power input from the first drive power generation unit 612 , thereby controlling the light output of the first illumination device 603 .
[0106] The first driving circuit 614 drives the first lighting device 603 under the control of the first controller 613 to output illumination light.
[0107] In the first illumination control section 601 configured in this manner, a first drive control section 611, a first drive power generation section 612, and a first controller 613 are provided in a primary circuit 6a. Also, a first drive circuit 614 is provided in a patient circuit 6b that is electrically insulated from the primary circuit 6a.
[0108] The second illumination control unit 602 includes a second drive control unit 621 , a second drive power generation unit 622 , a second controller 623 , and a second drive circuit 624 .
[0109] The second drive control unit 621 controls the driving of the second drive power generation unit 622 , the second controller 623 and the second drive circuit 624 .
[0110] The second drive power generation section 622 generates drive power for the second lighting device 604 under the control of the second drive control section 621 , and outputs this drive power to the second controller 623 .
[0111] The second controller 623 controls the second drive circuit 624 in accordance with the drive power input from the second drive power generation section 622, thereby controlling the light output of the second illumination device 604.
[0112] The second driving circuit 624 drives the second lighting device 604 under the control of the second controller 623 to output illumination light.
[0113] In second illumination control section 602 configured in this manner, second drive control section 621, second drive power generation section 622, and second controller 623 are provided in primary circuit 6a. Second drive circuit 624 is provided in patient circuit 6b that is electrically insulated from primary circuit 6a.
[0114] The first illumination device 603 irradiates the subject with light in the wavelength band of visible light (hereinafter simply referred to as "visible light") as first illumination light for illuminating the subject via the endoscope 201. Here, visible light refers to white light (wavelength band λ=380 nm to 780 nm). The first illumination device 603 is configured using, for example, a white LED (Light Emitting Diode) lamp or a halogen lamp.
[0115] The second illumination device 604 irradiates the subject with light in a wavelength band outside visible light (hereinafter simply referred to as "invisible light") as second illumination light for illuminating the subject via the endoscope 201. Here, the invisible light is infrared light (wavelength band λ=780 nm to 2500 nm). The second illumination device 604 is configured using, for example, an infrared LED lamp or the like.
[0116] The input unit 605 receives input of signals from each device constituting the treatment system 1, and outputs the received signals to the CPU 606 and the CPU 610 in the lighting circuit.
[0117] The CPU 606 and the lighting circuit CPU 610 work together to comprehensively control the operation of the lighting device 6. The CPU 606 reads a program stored in the memory 607 into a working area of the memory and executes it, and controls each component part through the execution of the program by the processor, whereby the hardware and software work together to control the operation of each part of the lighting device 6.
[0118] The memory 607 stores various information necessary for the operation of the lighting device 6 and various programs executed by the lighting device 6. The memory 607 is configured using RAM, ROM, and the like.
[0119] The wireless communication unit 608 is an interface for wireless communication with other devices, and is configured using a communication module capable of Wi-Fi, Bluetooth, or the like.
[0120] The communication interface 609 is an interface for communicating with the lighting circuit 6c.
[0121] The lighting circuit memory 630 stores various information and programs required for the operation of the first lighting device 603 and the second lighting device 604. The lighting circuit memory 630 is configured using RAM, ROM, and the like.
[0122] In the lighting device 6 configured in this manner, the input unit 605, CPU 606, memory 607, wireless communication unit 608, and communication interface 609 are provided in the primary circuit 6a. The first lighting device 603, the second lighting device 604, the lighting circuit CPU 610, and the lighting circuit memory 61A are provided in the lighting circuit 6c.
[0123] [Detailed functional configuration of the image processing unit] Next, the detailed functional configuration of the image processing unit 222 will be described. FIG. 10 is a block diagram showing the detailed functional configuration of the image processing unit 222. As shown in FIG. The image processing unit 222 shown in Figure 10 has an image data input unit 2221, an image generation unit 2222, a turbidity detection unit 2223, a turbidity determination unit 2224, a feature detection unit 2225, a corrected image generation unit 2226, an enhanced image generation unit 2227, a composite image generation unit 2228, a display image generation unit 2229, a memory 2230, and an image processing control unit 2231.
[0124] The image data input unit 2221 receives input of image data generated by the endoscope 201 and input of signals from each device constituting the treatment system 1, and outputs the received data and signals to the bus.
[0125] The image generation unit 2222 performs predetermined image processing on image data (RAW data) input via the image data input unit 2221 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204 to generate color image data (hereinafter simply referred to as "normal image data"), and outputs this normal image data to the turbidity detection unit 2223, the feature detection unit 2225, the corrected image generation unit 2226, and the display image generation unit 2229. Specifically, the image generation unit 2222 generates normal image data based on pixel values of R pixels, G pixels, and B pixels included in the image data. Here, examples of the predetermined image processing include demosaicing processing, color correction processing, black level correction processing, noise reduction processing, and gamma correction processing. In the first embodiment, the image generation unit 2222 functions as an image acquisition unit that acquires image data including an area to be treated on a living body by an energy treatment tool, for example, an ultrasound probe 312. The image generation unit 2222 may generate normal image data based on a drive signal of the treatment tool 301.
[0126] The turbidity detection unit 2223 detects a change in gradation from at least a partial region of a normal image (hereinafter simply referred to as a "normal image") corresponding to the normal image data, based on the normal image data generated by the image generation unit 2222, and outputs this detection result to the turbidity determination unit 2224. Specifically, the turbidity detection unit 2223 detects turbidity in the field of view of the endoscope 201 as at least a partial region of the normal image, based on the normal image generated by the image generation unit 2222, and outputs this detection result to the turbidity determination unit 2224. Note that the turbidity detection method by the turbidity detection unit 2223 is the same as the method for detecting turbidity components by a turbidity estimation unit 2226a of the corrected image generation unit 2226, which will be described later, and therefore a detailed description of the detection method will be omitted.
[0127] Here, the turbidity of the visual field of the endoscope 201 refers to the degree of turbidity due to bone powder and debris dissolved in the perfusion fluid, which is a factor in degrading gradation. Factors that degrade image quality include the dissolution of biological tissues, such as bone powder, debris, blood, and bone marrow, in the perfusion fluid, as well as smoke and sparks during treatment with the treatment tool 301. Below, turbidity is described as a state in which the perfusion fluid becomes cloudy when bone powder dissolves in the perfusion fluid. Perfusion fluid in which biological tissues have been dissolved becomes cloudy overall, and is characterized by high brightness, low saturation (low color reproduction), and low contrast. Therefore, the turbidity detection unit 2223 detects the turbidity (turbidity component) of the visual field of the endoscope 201 by calculating the contrast, brightness, and saturation for each pixel constituting the first image as the turbidity of the visual field of the endoscope 201.
[0128] The turbidity determination unit 2224 determines whether the turbidity detected by the turbidity detection unit 2223 is equal to or greater than a predetermined value, and outputs the determination result to the image processing control unit 2231. Here, the predetermined value is, for example, a value at a level where the treatment site disappears in the field of view of the endoscope 201 due to turbidity. For example, the level value that will eliminate the treatment area is a value of high brightness and low saturation (high brightness white).
[0129] The feature amount detection unit 2225 detects an area including at least a part of the treatment tool 301, which is an energy treatment tool, from the normal image generated by the image generation unit 2222, and outputs this detection result to the enhanced image generation unit 2227. The feature amount detection unit 2225 detects a feature amount for each pixel from the normal image generated by the image generation unit 2222, and detects an area including at least a part of the treatment tool 301, which is an energy treatment tool, based on this feature amount. Here, the feature amount is, for example, an edge component, a luminance component, etc. The feature amount detection unit 2225 may detect the treatment tool 301 appearing in the normal image using a well-known pattern matching processing technique.
[0130] The corrected image generation unit 2226 generates corrected image data by performing gradation correction processing on the normal image input from the image generation unit 2222 based on the detection result input from the muddyness detection unit 2223 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204, and outputs a corrected image corresponding to this corrected image data (hereinafter simply referred to as a "corrected image") to the enhanced image generation unit 2227, the composite image generation unit 2228, and the display image generation unit 2229. Specifically, the corrected image generation unit 2226 generates a corrected image by removing muddyness (mud component) contained in the normal image, and outputs this corrected image to the enhanced image generation unit 2227, the composite image generation unit 2228, or the display image generation unit 2229. Details of the corrected image generation unit 2226 will be described later.
[0131] The enhanced image generation unit 2227 performs edge enhancement processing on the normal image input from the image generation unit 2222 or the corrected image input from the corrected image generation unit 2226 based on a synchronization signal synchronized with the imaging drive of the imaging unit 204 and the detection result of the feature detection unit 2225, to generate enhanced image data, and outputs this enhanced image data (hereinafter simply referred to as "enhanced image") to the composite image generation unit 2228 or the display image generation unit 2229. Specifically, the enhanced image generation unit 2227 performs edge extraction processing to extract edge components on a region of the normal image or the corrected image including at least a part of the treatment tool 301, which is an energy treatment tool, detected by the feature detection unit 2225, and generates an enhanced image by performing edge enhancement processing on the extracted edge components to enhance the edges compared to other regions.
[0132] Under the control of the image processing control unit 2231, the composite image generation unit 2228 generates composite image data by combining the normal image input from the image generation unit 2222 or the corrected image input from the corrected image generation unit 2226 with the enhanced image input from the enhanced image generation unit 2227 at a predetermined ratio in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204, and outputs a composite image corresponding to this composite image data (hereinafter simply referred to as the "composite image") to the display image generation unit 2229. Here, the predetermined ratio is, for example, 5:5. The composite image generation unit 2228 may change the ratio at which the normal image or the corrected image and the enhanced image are combined based on the detection result of the turbidity detection unit 2223, or may change the combination ratio as appropriate, and may also change the combination ratio depending on the component and type of turbidity. Furthermore, the composite image may be generated by adding or superimposing an edge component extracted from the enhanced image on the normal image or the corrected image.
[0133] The display image generation unit 2229, under the control of the image processing control unit 2231, generates a display image corresponding to display image data to be displayed on the display device 203 based on one or more of the normal image input from the image generation unit 2222, the corrected image input from the corrected image generation unit 2226, the enhanced image input from the enhanced image generation unit 2227, and the composite image input from the composite image generation unit 2228 in accordance with a synchronization signal synchronized with the imaging drive of the imaging unit 204, and outputs the generated display image to the display device 203. Specifically, the display image generation unit 2229 converts, for example, the format of the input image into a predetermined format, for example, converting the RGB format into the YCbCr format, and outputs the converted image to the display device 203. The display image generation unit 2229 may generate a display image based on a drive signal of the treatment tool 301.
[0134] The memory 2230 stores various types of information necessary for the operation of the image processing unit 222, various programs executed by the image processing unit 222, various types of image data, etc. The memory 2230 is configured using RAM, ROM, frame memory, etc.
[0135] The image processing control unit 2231 controls each unit constituting the image processing unit 222. The image processing control unit 2231 reads a program stored in the memory 2230 into a working area of the memory and executes it, and controls each component unit etc. through the execution of the program by the processor, thereby controlling the operation of each unit constituting the image processing unit 222 through cooperation between hardware and software.
[0136] [Detailed Functional Configuration of Corrected Image Generation Unit] Next, the detailed mechanical configuration of the corrected image generating unit 2226 will be described. FIG. 11 is a block diagram showing the detailed functional configuration of the corrected image generating unit 2226. As shown in FIG.
[0137] The corrected image generating unit 2226 shown in FIG. 11 includes a muddyness estimating unit 2226a, a histogram generating unit 2226b, a representative brightness calculating unit 2226c, a correction coefficient calculating unit 2226d, and a contrast correcting unit 2226e.
[0138] The turbidity estimation unit 2226a estimates the turbidity component for each pixel in the normal image. Here, the turbidity component for each pixel refers to the degree of turbidity due to bone powder and debris dissolved in the perfusion fluid, which is a factor that degrades image quality such as contrast and saturation in the normal image. Factors that degrade image quality include the dissolution of biological tissues such as bone powder, debris, blood, and bone marrow in the perfusion fluid, as well as smoke and sparks during treatment with the treatment tool 301. Below, turbidity will be described as the turbidity that occurs when bone powder dissolves in the perfusion fluid and the fluid becomes cloudy. Perfusion fluid in which biological tissues have been dissolved is characterized by high brightness, low saturation (low color reproduction), and low contrast.
[0139] For this reason, the turbidity estimation unit 2226a calculates the contrast or brightness and saturation of the first image to estimate the turbidity component of the field of view of the endoscope 201. Specifically, the turbidity estimation unit 2226a estimates the turbidity component H(x, y) based on the R value, G value, and B value of the pixel at the coordinates (x, y) in the first image.
[0140] Here, if the R value, G value, and B value at coordinates (x, y) are Ir, Ig, and Ib, respectively, the turbidity component H(x, y) of the pixel at coordinates (x, y) is estimated by the following equation (1). H(x,y)=min(Ir,Ig,Ib) ···(1)
[0141] The turbidity estimation unit 2226a performs the calculation of the above-mentioned formula (1) for each pixel of the first image. The turbidity estimation unit 2226a sets a scan area F (small area) of a predetermined size for the first image. The size of this scan area F is, for example, a predetermined size of m×n pixels (m and n are natural numbers). In the following description, the pixel at the center of the scan area F will be referred to as the reference pixel. Furthermore, in the following description, each pixel around the reference pixel in the scan area F will be referred to as a neighboring pixel. Furthermore, in the following description, the scan area F will be described as being formed to a size of, for example, 5×5 pixels. Of course, the scan area F can also be applied if it is just one pixel.
[0142] The turbidity estimation unit 2226a calculates (Ir, Ig, Ib) for each pixel in the scan area F while shifting the position of the scan area F relative to the first image, and estimates the minimum value among them as the turbidity component H(x, y) of the reference pixel. Pixel values in high-brightness, low-saturation areas in the first image have similar and large R, G, and B values, so the value of min(Ir, Ig, Ib) is large. That is, areas with high brightness and low saturation have a large value for the turbidity component H(x, y).
[0143] In contrast, pixel values in low brightness or high saturation areas have small R, G, or B values, so the value of min(Ir, Ig, Ib) is small. That is, in low brightness or high saturation areas, the turbidity component H(x, y) has a small value.
[0144] Thus, the turbidity component H(x,y) increases as the concentration of bone powder dissolved in the perfusion fluid increases (as the white color of the bone powder increases), and decreases as the concentration of bone powder dissolved in the perfusion fluid decreases. In other words, the turbidity component H(x,y) increases as the color (white) of the perfusion fluid increases due to the bone powder dissolved in the perfusion fluid, and decreases as the color of the perfusion fluid decreases.
[0145] The turbidity estimation unit 2226a estimates the turbidity component H(x, y) using the above-mentioned formula (1), but is not limited to this, and any index showing high brightness and low saturation can be used as the turbidity component. The turbidity estimation unit 2226a may estimate the turbidity component using one or more of the local contrast value, edge intensity, color density, and object distance. Furthermore, the above-mentioned turbidity detection unit 2223 detects turbidity (turbidity component) using the same method as the turbidity estimation unit 2226a.
[0146] The histogram generation unit 2226b determines the distribution of a histogram in a local region including a reference pixel of the first image and neighboring pixels around the reference pixel, based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a. The degree of change in this turbidity component (x, y) serves as an index for determining the region to which each pixel belongs in the local region. Specifically, the degree of change in this turbidity component (x, y) is determined based on the difference in the turbidity component H(x, y) between the reference pixel and neighboring pixels in the local region.
[0147] That is, the histogram generation unit 2226b generates a brightness histogram for a local region including neighboring pixels for each reference pixel based on the first image input from the image generation unit 2222 and the turbidity component H(x, y) input from the turbidity estimation unit 2226a. A general histogram is generated by regarding pixel values in the target local region as brightness values and counting the frequency of pixel values one by one.
[0148] In contrast, the histogram generating unit 2226b according to the first embodiment weights the count value for the pixel value of a neighboring pixel according to the turbidity component H(x,y) between the reference pixel and the neighboring pixel in the local region. The count value for the pixel value of the neighboring pixel is, for example, a value in the range of 0.0 to 1.0. The count value is set so that the larger the difference between the turbidity component H(x,y) between the reference pixel and the neighboring pixel is, the smaller the value becomes, and the smaller the difference between the turbidity component H(x,y) between the reference pixel and the neighboring pixel is, the larger the value becomes. Furthermore, the local region is formed to have a size of, for example, 7x7 pixels.
[0149] In general, if a histogram is generated based only on luminance, the luminance of neighboring pixels with large differences from the luminance of the pixel of interest will also be counted. Therefore, it is desirable to generate a local histogram based on the image region to which the pixel of interest belongs.
[0150] In contrast, in generating a brightness histogram in the first embodiment, a count value for the pixel value of each pixel in a local region in the first image data is set according to the difference in the turbidity component H(x,y) between a reference pixel and each neighboring pixel in the local region in the first image data. Specifically, the count value is calculated using, for example, a Gaussian function so that the larger the difference in the turbidity component H(x,y) between the reference pixel and the neighboring pixels, the smaller the count value, and the smaller the difference in the turbidity component H(x,y) between the reference pixel and the neighboring pixels, the larger the count value (see, for example, Japanese Patent No. 6720012 or Japanese Patent No. 6559229, where the haze component is replaced with the turbidity component).
[0151] The method of calculating the count value by the histogram generation unit 2226b is not limited to a Gaussian function, and any method may be used as long as the count value decreases as the difference between the values of the reference pixel and the neighboring pixels increases. For example, the histogram generation unit 2226b may calculate the count value using a lookup table or a table approximated by a broken line instead of a Gaussian function.
[0152] The histogram generating unit 2226b may also compare the difference in value between the reference pixel and the neighboring pixel with a threshold, and if the difference is equal to or greater than the threshold, reduce the count value of the neighboring pixel (for example, set it to 0.0).
[0153] Furthermore, the histogram generation unit 2226b does not necessarily have to use the frequency of pixel values as count values. For example, the histogram generation unit 2226b may use each of the R value, G value, and B value as count values. Furthermore, the histogram generation unit 2226b may use the G value as a luminance value as count value.
[0154] The representative luminance calculation unit 2226c calculates a representative luminance based on statistical information of the luminance histogram input from the histogram generation unit 2226b. The representative luminance is the luminance of the low luminance part, the luminance of the high luminance part, and the luminance of the intermediate luminance part of the effective luminance range of the luminance histogram. The luminance of the low luminance part is the minimum luminance of the effective luminance range. The luminance of the high luminance part is the maximum luminance of the effective luminance range. The luminance of the intermediate luminance part is the centroid luminance. The minimum luminance is the luminance at which the cumulative frequency is 5% of the maximum value in a cumulative histogram created from the luminance histogram. The maximum luminance is the luminance at which the cumulative frequency is 95% of the maximum value in a cumulative histogram created from the luminance histogram. The centroid luminance is the luminance at which the cumulative frequency is 50% of the maximum value in a cumulative histogram created from the luminance histogram.
[0155] The percentages of cumulative frequency corresponding to the minimum luminance, maximum luminance, and centroid luminance, 5%, 50%, and 95%, can be changed as appropriate. Furthermore, the luminance of the intermediate luminance portion is set to the centroid luminance in the cumulative histogram, but this is not limited thereto, and the centroid luminance does not necessarily have to be calculated from the cumulative frequency. For example, the luminance of the intermediate luminance portion can also be set to the luminance with the highest frequency in the luminance histogram.
[0156] The correction coefficient calculation unit 2226d calculates a correction coefficient for correcting the contrast in a local region based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the statistical information input from the representative luminance calculation unit 2226c. Specifically, when contrast correction is performed by histogram expansion, the correction coefficient calculation unit 2226d calculates a coefficient for histogram expansion using the centroid luminance and maximum luminance from the statistical information.
[0157] Here, histogram expansion is a process that enhances contrast by expanding the effective luminance range of the histogram (see, for example, Japanese Patent No. 6720012 or Japanese Patent No. 6559229). Note that the correction coefficient calculation unit 2226d uses histogram expansion as a means of achieving contrast correction, but is not limited to this. Histogram flattening, for example, may also be applied as a means of achieving contrast correction. For example, the correction coefficient calculation unit 2226d may use a cumulative histogram or a table that approximates a broken line as a method of achieving histogram flattening. This cumulative histogram is obtained by sequentially accumulating the most frequent values of the luminance histogram.
[0158] The contrast correction unit 2226e performs contrast correction of the reference pixel of the first image data for the first image input from the image generation unit 2222 based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the correction coefficient input from the correction coefficient calculation unit 2226d (see, for example, Patent Publication No. 6720012 or Patent Publication No. 6559229).
[0159] The corrected image generation unit 2226 configured in this way estimates the turbidity component H(x, y) based on the first image, calculates a brightness histogram and representative brightness using this estimation result, calculates a correction coefficient for correcting contrast within a local region, and performs contrast correction based on the turbidity component H(x, y) and the correction coefficient. In this way, the corrected image generation unit 2226 can generate a first corrected image in which turbidity has been removed from the first image.
[0160] [Summary of treatment] Next, an outline of the treatment performed by the surgeon using the treatment system 1 will be described. FIG. 12 is a flowchart outlining a procedure performed by an operator using the treatment system 1. As shown in FIG. The operator performing the procedure may be a single doctor, or two or more including a doctor and an assistant.
[0161] As shown in FIG. 12, first, the surgeon forms a first portal P1 and a second portal P2 that respectively connect the inside of the joint cavity C1 of the knee joint J1 with the outside of the skin (step S1).
[0162] Next, the surgeon inserts the endoscope 201 into the joint cavity C1 through the first portal P1, inserts the guiding device 4 into the joint cavity C1 through the second portal P2, and inserts the treatment tool 301 into the joint cavity C1 under the guidance of the guiding device 4 (step S2). Note that, although the case where two portals are formed and then the endoscope 201 and the treatment tool 301 are inserted into the joint cavity C1 through the first portal P1 and the second portal P2 has been described here, it is also possible to form the first portal P1 and insert the endoscope 201 into the joint cavity C1, and then form the second portal P2 and insert the guiding device 4 and the treatment tool 301 into the joint cavity C1.
[0163] Thereafter, the surgeon brings the ultrasonic probe 312 into contact with the bone to be treated while visually checking the endoscopic image of the inside of the joint cavity C1 displayed on the display device 203 (step S3).
[0164] Next, the surgeon performs ablation treatment using the treatment tool 301 while viewing the endoscopic image displayed on the display device 203 (step S4). Details of the processing of the treatment system 1 during ablation treatment will be described later.
[0165] Thereafter, the display device 203 performs a display and notification process for the inside of the joint cavity C1 and information regarding the state after the cutting treatment (step S5). After the display and notification process, the endoscope control device 202 is turned off, for example. The surgeon ends the treatment using the treatment system 1.
[0166] [Details of cutting procedure] Next, the cutting treatment in step S4 of FIG. 12 will be described in detail. FIG. 13 provides an overview of the processing executed by the endoscope control device 202 during ablation treatment. In the following description, each process is executed under the control of the CPU of each control device, but the processes may be executed collectively by any one of the control devices such as the network control device 7, for example.
[0167] As shown in FIG. 13, the CPU 227 communicates with each device, sets control parameters for each of the treatment device 3 and the perfusion device 5, and inputs the control parameters for each of the treatment device 3 and the perfusion device 5 (step S11).
[0168] Next, the CPU 227 determines whether or not the devices of the parts constituting the treatment system 1 have entered an output ON state (step S12). If the CPU 227 determines that the devices of the parts constituting the treatment system 1 have entered an output ON state (step S12: Yes), the endoscope control device 202 proceeds to step S13, which will be described later. On the other hand, if the CPU 227 determines that the devices of the parts constituting the treatment system 1 have not entered an output ON state (step S12: No), the CPU 227 continues this determination until the devices of the parts constituting the treatment system 1 enter an output ON state.
[0169] In step S13, the CPU 227 determines whether the observation mode of the endoscope control device 202 in the treatment system 1 is set to the turbidity detection mode. If the CPU 227 determines that the observation mode of the endoscope control device 202 in the treatment system 1 is set to the turbidity detection mode (step S13: Yes), the endoscope control device 202 proceeds to step S14, which will be described later. On the other hand, if the CPU 227 determines that the observation mode of the endoscope control device 202 in the treatment system 1 is not set to the turbidity detection mode (step S13: No), the endoscope control device 202 proceeds to step S16, which will be described later.
[0170] In step S14, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 based on any one of the normal image generated by the endoscope 201, the detection result of the impedance detection unit 330 of the treatment instrument control device 302, and the detection result of the turbidity detection unit 516 of the perfusion device 5. Specifically, when the normal image generated by the endoscope 201 is used, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 using either the brightness or contrast of the normal image. Furthermore, when the impedance detected by the impedance detection unit 330 of the treatment instrument control device 302 is used, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 based on the rate of change of the impedance. Furthermore, when the detection result of the turbidity detection unit 516 of the perfusion device 5 is used, the turbidity detection unit 223 detects turbidity in the field of view of the endoscope 201 based on the turbidity of the perfusion fluid detected by the turbidity detection unit 516 of the perfusion device 5.
[0171] Next, the CPU 227 determines whether or not the turbidity in the field of view of the endoscope 201 is equal to or greater than a predetermined value based on the detection result detected by the turbidity detection unit 223 (step S15).
[0172] Specifically, when the turbidity detection unit 223 uses a normal image, the CPU 227 determines whether the average value of the sum of the brightness values of each pixel of the normal image detected by the turbidity detection unit 223 is equal to or greater than a predetermined value. Here, the predetermined value for brightness is a high saturation value that is as close to white as possible. In this case, the CPU 227 determines that turbidity has occurred in the field of view of the endoscope 201 when the average value of the sum of the brightness values of each pixel of the normal image detected by the turbidity detection unit 223 is equal to or greater than the predetermined value. On the other hand, the CPU 227 determines that turbidity has not occurred in the field of view of the endoscope 201 when the average value of the brightness values of each pixel of the normal image detected by the turbidity detection unit 223 is equal to or greater than the predetermined value.
[0173] Furthermore, when the turbidity detection unit 223 uses the impedance detected by the impedance detection unit 330, the CPU 227 determines whether the impedance is equal to or greater than a predetermined value. When the impedance detected by the impedance detection unit 330 by the turbidity detection unit 223 is equal to or greater than a predetermined value, the CPU 227 determines that turbidity has occurred in the field of view of the endoscope 201. On the other hand, when the impedance detected by the impedance detection unit 330 is not equal to or greater than the predetermined value, the turbidity detection unit 223 determines that turbidity has not occurred in the field of view of the endoscope 201.
[0174] Furthermore, when the turbidity detection unit 223 uses the turbidity of the perfusion fluid detected by the turbidity detection unit 516 of the perfusion device 5, the CPU 227 determines whether the turbidity of the perfusion fluid is equal to or greater than a predetermined value. When the turbidity of the perfusion fluid detected by the turbidity detection unit 223 is equal to or greater than the predetermined value, the CPU 227 determines that turbidity has occurred in the field of view of the endoscope 201. On the other hand, when the turbidity of the perfusion fluid detected by the turbidity detection unit 223 is not equal to or greater than the predetermined value, the CPU 227 determines that turbidity has not occurred in the field of view of the endoscope 201.
[0175] In step S15, if the CPU 227 determines that the field of view of the endoscope 201 is cloudy (step S15: Yes), the endoscope control device 202 proceeds to step S19, which will be described later. On the other hand, if the CPU 227 determines that the field of view of the endoscope 201 is not cloudy (step S15: No), the endoscope control device 202 proceeds to step S16, which will be described later.
[0176] In step S16, the CPU 227 performs normal control over the endoscope control device 202. Specifically, the CPU 227 outputs the normal image (color image) generated by the image processing unit 222 to the display device 203 for display. This allows the surgeon to perform treatment using the treatment tool 301 while viewing the normal image displayed on the display device 203.
[0177] Next, the CPU 227 determines whether the surgeon is continuing the treatment on the subject (step S17). Specifically, the CPU 227 determines whether the treatment instrument control device 302 is supplying power to the treatment instrument 301. If the treatment instrument control device 302 is supplying power to the treatment instrument 301, the CPU 227 determines that the surgeon is continuing the treatment on the subject. If the treatment instrument control device 302 is not supplying power to the treatment instrument 301, the CPU 227 determines that the surgeon is not continuing the treatment on the subject. If the CPU 227 determines that the surgeon is continuing the treatment on the subject (step S17: Yes), the endoscope control device 202 proceeds to step S18, which will be described later. On the other hand, if the CPU 227 determines that the surgeon is not continuing the treatment on the subject (step S17: No), the endoscope control device 202 ends this process.
[0178] In step S18, the CPU 227 determines whether or not the output of each device constituting the treatment system 1 has been turned off. If the CPU 227 determines that the output of each device constituting the treatment system 1 has been turned off (step S18: Yes), the endoscope control device 202 ends this processing. On the other hand, if the CPU 227 determines that the output of each device constituting the treatment system 1 has not been turned off (step S18: No), the endoscope control device 202 returns to step S13 described above.
[0179] In step S19, the endoscope control device 202 executes turbidity response control processing for turbidity in the field of view of the endoscope 201. Details of the turbidity response control processing will be described later. After step S19, the endoscope control device 202 proceeds to step S17.
[0180] [Details of Turbidity Control Processing] Next, the turbidity response control process described in step S19 of FIG. 13 will be described in detail. FIG. 14 is a flowchart showing a detailed outline of the turbidity response control process of FIG.
[0181] 14, the image generation unit 2222 generates a normal image (step S101). Specifically, the image generation unit 2222 generates a normal image (a color image using visible light) based on image data input from the image data input unit 2221.
[0182] Next, the turbidity detection unit 2223 estimates the turbidity of the field of view of the endoscope 201 based on the normal image generated by the image generation unit 2222, and the feature detection unit 2225 detects a region including at least a portion of the treatment tool 301, which is an energy treatment tool, from the normal image (step S102). Specifically, the turbidity detection unit 2223 estimates the turbidity component of the field of view of the endoscope 201 using an estimation method similar to that used by the turbidity estimation unit 2226a described above. Furthermore, the feature detection unit 2225 detects a region including at least a portion of the treatment tool 301 by performing edge extraction processing on the normal image to extract edge components. Of course, the feature detection unit 2225 may also perform luminance extraction processing on the normal image to extract luminance components, and detect a region with a luminance value equal to or greater than a predetermined value as a region including at least a portion of the treatment tool 301. Furthermore, the edge extraction processing may be performed using a combination of at least one of well-known filters, such as a Sobel filter, a Laplacian filter, and a Canny filter.
[0183] Thereafter, the turbidity determination unit 2224 determines whether the turbidity of the field of view of the endoscope 201 detected by the turbidity detection unit 2223 is equal to or greater than a predetermined value (step S103). If the turbidity determination unit 2224 determines that the turbidity component of the field of view of the endoscope 201 detected by the turbidity detection unit 2223 is equal to or greater than a predetermined value (step S103: Yes), the endoscope control device 202 proceeds to step S104, which will be described later. On the other hand, if the turbidity determination unit 2224 determines that the turbidity component of the field of view of the endoscope 201 detected by the turbidity detection unit 2223 is not equal to or greater than the predetermined value (step S103: No), the endoscope control device 202 proceeds to step S109, which will be described later.
[0184] In step S104, the corrected image generation unit 2226 generates a corrected image by performing gradation correction to remove or reduce turbidity on the normal image input from the image generation unit 2222, based on the detection result input from the turbidity detection unit 2223. After step S104, the endoscope control device 202 proceeds to step S105, which will be described later.
[0185] Fig. 15 is a diagram showing an example of a displayed image when the field of view of the endoscope 201 is good. Fig. 16 is a diagram showing the relationship between the position on the line AA' in Fig. 15 and the brightness. FIG. 17 is a diagram showing an example of a display image when the field of view of the endoscope 201 is poor. FIG. 18 is a diagram showing the relationship between the position on the line AA' in FIG. 16 and the luminance. FIG. 19 is a diagram showing the relationship between the position on the line AA' and the brightness after the corrected image generating unit 2226 has performed gradation correction on the display image of FIG. 16, 18, and 19, the horizontal axis represents position on the displayed image, and the vertical axis represents brightness. Curve L1 in Fig. 16 represents the relationship between brightness and position on the displayed image, curve L2 in Fig. 18 represents the relationship between brightness and position on the displayed image, and curve L3 in Fig. 19 represents the relationship between brightness and position on the displayed image.
[0186] 15 to 19, when the field of view of the endoscope 201 changes from a good state to a bad state, the field of view of the endoscope 201 becomes cloudy due to bone powder or the like (see display image Q1 in FIG. 15 → display image Q2 in FIG. 17), and brightness increases even in areas other than those in which the ultrasonic probe 312 or the like is captured (curve L1 in FIG. 16 → curve L2 in FIG. 18). For this reason, the corrected image generation unit 2226 generates a corrected image by performing gradation correction on the normal image input from the image generation unit 2222 to remove or reduce cloudiness, based on the detection result input from the cloudiness detection unit 2223. In this case, as shown by curve L3 in FIG. 19, the corrected image generation unit 2226 generates a corrected image so that the brightness is increased in areas in which at least the ultrasonic probe 312 and the treatment target site 100 are captured, and the brightness is decreased in other areas.
[0187] Returning to FIG. 14, the description of step S105 and subsequent steps will be continued. In step S105, the enhanced image generation unit 2227 generates an enhanced image by performing edge enhancement processing on the corrected image generated by the corrected image generation unit 2226 based on the detection result detected by the feature detection unit 2225. After step S105, the endoscope control device 202 proceeds to step S106, which will be described later.
[0188] FIG. 20 is a diagram showing the relationship between the position and brightness of the emphasized image generated by the emphasized image generating unit 2227. 20 illustrates the relationship between the position and the luminance on the same line A-A' as in FIG. 15. In FIG. 20, the horizontal axis indicates the position in the enhanced image, and the vertical axis indicates the luminance. Furthermore, a curve L4 in FIG. 20 indicates the relationship between the luminance and the position on the enhanced image.
[0189] 20, the enhanced image generating unit 2227 performs edge enhancement processing on the region detected by the feature amount detecting unit 2225 included in the corrected image generated by the corrected image generating unit 2226, thereby generating an enhanced image in which the edge components are enhanced compared to other regions. Specifically, the enhanced image generating unit 2227 performs edge enhancement processing so as to enhance the outline (enhanced region) of at least the region in which the ultrasound probe 312 and the treatment target region 100 are shown, thereby generating an enhanced image. In addition to the edge enhancement processing, the enhanced image generating unit 2227 may generate an enhanced image in which the outline of the region in which the treatment target region 100 is shown is emphasized, for example, by a color designated by the surgeon, such as red or green.
[0190] Returning to FIG. 14, the description of step S106 and subsequent steps will be continued. In step S106, the composite image generation unit 2228 generates a composite image by combining the normal image generated by the image generation unit 2222 with at least one of the corrected image generated by the corrected image generation unit 2226 and the enhanced image generated by the enhanced image generation unit 2227. After step S106, the endoscope control device 202 proceeds to step S107, which will be described later.
[0191] FIG. 21 is a diagram showing a schematic diagram of a method for generating a composite image by the composite image generating unit 2228. In FIG. As shown in FIG. 21, the composite image generating unit 2228 generates a composite image Q14 by combining a normal image Q11, a corrected image Q12, and an enhanced image Q13 at a predetermined ratio.
[0192] In step S107, the display image generation unit 2229 generates a display image based on the composite image generated by the composite image generation unit 2228 and outputs the display image to the display device 203. Fig. 22 is a diagram showing an example of a display image displayed by the display device 203. As shown in Fig. 22, the outline of the treatment tool 301 shown in part of the display image Q21 is emphasized, so the surgeon can perform cutting on the treatment target area 100 with the ultrasonic probe 312 without interruption. After step S107, the endoscope control device 202 returns to the main routine for cutting treatment in Fig. 13 and proceeds to step S17.
[0193] In step S108, the display image generation unit 2229 generates a display image based on the normal image generated by the image generation unit 2222, and outputs the display image to the display device 203. After step S108, the endoscope control device 202 returns to the main routine for the cutting treatment in Fig. 13, and proceeds to step S17.
[0194] According to the above-described embodiment 1, the display image generation unit 2229 generates a display image based on the enhanced image input from the enhanced image generation unit 2227 and outputs the display image to the display device 203. Therefore, even if the field of view of the endoscope 201 deteriorates, the position of the treatment tool 301 can be grasped, and treatment of the treatment target area 100 can be continued.
[0195] Furthermore, according to the first embodiment, the composite image generating unit 2228 generates a composite image by combining the corrected image generated by the corrected image generating unit 2226 and the emphasized image generated by the emphasized image generating unit 2227, and the display image generating unit 2229 generates a display image based on the composite image and outputs it to the display device 203. As a result, even if the field of view of the endoscope 201 deteriorates, the surgeon can grasp the position of the treatment tool 301, and can continue the treatment on the treatment target site 100.
[0196] Furthermore, according to the first embodiment, when the turbidity determination unit 2224 determines that the turbidity is equal to or greater than a predetermined value, the display image generation unit 2229 generates a display image based on the enhanced image input from the enhanced image generation unit 2227 and outputs the display image to the display device 203, whereas when the turbidity determination unit 2224 determines that the turbidity is not equal to or greater than the predetermined value, the display image generation unit 2229 outputs a display image based on the normal image generated by the image generation unit 2222. This allows the surgeon to perform treatment while viewing the display image, which is the normal image, when the field of view of the endoscope 201 is good, whereas when the field of view of the endoscope 201 is poor, the surgeon can perform treatment while viewing the display image in which the treatment tool 301 is enhanced.
[0197] In the first embodiment, the composite image generation unit 2228 may generate a composite image by combining the normal image generated by the image generation unit 2222 and the enhanced image generated by the enhanced image generation unit 2227, and the display image generation unit 2229 may generate a display image based on the composite image and output it to the display device 203.
[0198] (Embodiment 2) Next, a second embodiment will be described. In the first embodiment described above, the composite image generation unit 2228 generates a composite image by combining a normal image or a corrected image with an enhanced image. However, in the second embodiment, a composite image is generated by combining the position of the treatment tool in the normal image before the occurrence of turbidity with the position of the treatment tool in the corrected image, and output to the display device. Specifically, the configuration of the image processing unit and the turbidity response control process according to the second embodiment are different. Therefore, the configuration of the image processing unit according to the second embodiment will be described below, and then the turbidity response control process executed by the endoscope control device 202 according to the second embodiment will be described.
[0199] [Functional configuration of image processing unit] FIG. 23 is a block diagram illustrating a functional configuration of an image processing unit according to the second embodiment. The image processing unit 222A shown in FIG. 23 further includes a position detection unit 2232 in addition to the configuration of the image processing unit 222 according to the first embodiment described above. Furthermore, the image processing unit 222A includes a composite image generating unit 2228A instead of the composite image generating unit 2228 according to the first embodiment described above.
[0200] The position detection unit 2232 detects a part of the treatment tool 301 that appears in the normal image generated by the image generation unit 2222. Specifically, the position detection unit 2232 detects a feature amount for each pixel between the normal image generated by the image generation unit 2222 or the corrected image generated by the corrected image generation unit 2226 and the enhanced image generated by the enhanced image generation unit 2227, and detects a matching portion based on this feature amount. Here, the feature amount is, for example, an edge component and a luminance component. The position detection unit 2232 may detect the treatment tool 301 that appears in the normal image using a well-known pattern matching processing technique and output this detection result to the composite image generation unit 2228A.
[0201] Based on the detection result detected by the position detection unit 2232, the composite image generation unit 2228A generates a composite image by combining the normal image stored in the memory 2230 before the turbidity was detected and the corrected image generated by the corrected image generation unit 2226.
[0202] [Details of Turbidity Control Processing] Next, the turbidity response control process executed by the endoscope control device 202 according to the second embodiment will be described in detail. FIG. 24 is a flowchart showing a detailed outline of the turbidity response control process that is executed by the CPU 227 of the endoscope control device 202 according to the second embodiment issuing instructions to each component via the BUS.
[0203] Steps S201 to S204 correspond to steps S101 to S104 in FIG. 14, respectively.
[0204] The position detection unit 2232 detects the position of a match between the normal image generated by the image generation unit 2222 or the corrected image generated by the corrected image generation unit 2226 and the enhanced image generated by the enhanced image generation unit 2227 (step S205).
[0205] Next, the composite image generating unit 2228A executes a synthesis process to generate a composite image by synthesizing the normal image before the turbidity was detected and stored in the memory 2230 and the corrected image generated by the corrected image generating unit 2226, based on the detection result detected by the position detecting unit 2232 (step S206). Here, the normal image before the turbidity was detected and stored in the memory 2230 is used as an emphasized image in the second embodiment.
[0206] FIG. 25 is a diagram showing a schematic diagram of a method for generating a composite image by the composite image generating unit 2228A. As shown in FIG. 25, when the treatment tool 301 is driven (PW on) and the temporally consecutive normal images (normal images Q31 to Q33) that are the field of view of the endoscope 201 gradually become cloudy and turn into normal images with poor field of view (normal images Q34 to Q38), the composite image generation unit 2228A sequentially generates composite images (composite images Q41 to Q43) by combining the normal image (for example, normal image Q33) before the cloudiness was detected that is stored in the memory 2230 and the corrected image generated by the corrected image generation unit 2226 based on the detection result detected by the position detection unit 2232. Specifically, the composite image generation unit 2228A sequentially generates composite images (composite images Q41 to Q43) by combining a normal image (e.g., normal image Q33) before turbidity is detected that is stored in the memory 2230 at any one of the time when the treatment of the treatment target area 100 with the treatment tool 301 starts, immediately after the start of the treatment, and before the start of the treatment and the corrected image generated by the corrected image generation unit 2226, based on the drive signal of the treatment tool 301. As a result, even if the field of view of the endoscope 201 becomes poor, the surgeon can perform cutting of the treatment target area 100 with the treatment tool 301 without interruption, because the contour of the treatment tool 301 is emphasized while viewing a smooth display image displayed by the display device 203.
[0207] After step S206, the display image generation unit 2229 generates a display image based on the composite image generated by the composite image generation unit 2228 and outputs the display image to the display device 203 (step S207). After step S206, the endoscope control device 202 returns to the main routine of the cutting treatment in Fig. 13 and proceeds to step S17.
[0208] In step S208, the image processing control unit 2231 records the image data, which is the normal image generated by the image generation unit 2222, in the memory 2230.
[0209] Next, the display image generation unit 2229 generates a display image based on the normal image generated by the image generation unit 2222 and outputs it to the display device 203 (step S209). After step S209, the endoscope control device 202 returns to the main routine of the cutting treatment in Fig. 13 and proceeds to step S17.
[0210] According to the above-described embodiment 2, the same effects as those of the above-described embodiment 1 are achieved, and even if the field of view of the endoscope 201 deteriorates, treatment of the treatment target area 100 by the treatment tool 301 can be continued.
[0211] (Embodiment 3) Next, a third embodiment will be described. In the third embodiment, a mark is provided near the tip of the treatment tool, the mark provided near the tip of the treatment tool is detected, and an enhanced image is generated based on the detection result. Specifically, in the third embodiment, the configurations of the image processing unit and the treatment tool are different. The configurations of the image processing unit and the treatment tool according to the third embodiment will be described below. Note that the same components as those in the treatment system 1 according to the first embodiment described above are assigned the same reference numerals, and detailed description thereof will be omitted.
[0212] [Functional configuration of image processing unit] FIG. 26 is a block diagram illustrating a functional configuration of an image processing unit according to the third embodiment. 26 includes an HDR (High Dynamic Range) image generation unit 2233 in addition to the configuration of the image processing unit 222 according to the above-described first embodiment. Furthermore, it includes a composite image generation unit 2228B instead of the composite image generation unit 2228 according to the above-described first embodiment.
[0213] The HDR image generation unit 2233 generates an HDR image using two images with a difference in brightness, and outputs this HDR image to the display image generation unit 2229. Specifically, the HDR image is generated using two normal images with a difference in brightness.
[0214] The enhanced image generating unit 2227 detects an image area corresponding to a mark near the tip of the treatment tool from the HDR image, and generates an enhanced image that has been subjected to edge enhancement processing according to the degree of edge visibility in the image data. The composite image generating unit 2228B generates a composite image by combining the HDR image generated by the HDR image generating unit 2233 with at least one of the corrected image generated by the corrected image generating unit 2226 and the enhanced image generated by the enhanced image generating unit 2227.
[0215] [Configuration of processing tool] FIG. 27 is a schematic diagram showing a schematic configuration of a part of a treatment tool according to the third embodiment. 27 includes a retroreflecting portion 320 that functions as an indicator and is provided on the proximal side of the ultrasonic transducer 312a along the longitudinal direction. The retroreflecting portion 320 reflects illumination light emitted from the endoscope 201. The retroreflecting portion 320 is configured using a retroreflecting member or the like.
[0216] The image processing unit 222B configured in this manner executes the same turbidity response control processing as in the above-described embodiments 1 and 2. In this case, the normal image corresponding to the field of view of the endoscope 201 is blown out due to the light reflected from the retroreflector 320 on the treatment tool 301B, and exceeds the dynamic range of the image sensor 2241. Therefore, the HDR image generation unit 2233 generates an HDR image using two images with a difference in brightness, thereby preventing the image from being blown out, even when the treatment tool 301 is provided with the retroreflector 320.
[0217] FIG. 28 is a diagram showing an example of a display image in which a part of the treatment tool 301B is displayed in the field of view of the endoscope 201. In FIG. FIG. 29 is a diagram showing the relationship between the position on the line AA' in FIG. 28 and the luminance. FIG. 30 is a diagram showing the relationship between the position on the same line as the line AA' in FIG. 28 and the luminance in the HDR image generated by the HDR image generating unit 2233. FIG. 31 is a diagram showing the relationship between the position on the same line as the line AA' in FIG. 28 and the brightness in a displayed image when the field of view of the endoscope 201 is poor. FIG. 32 is a diagram showing the relationship between the position on the same line as the line AA' in FIG. 28 and the brightness in the composite image generated by the composite image generating unit 2228A. 29 to 32, the horizontal axis indicates the position on the image, and the vertical axis indicates the brightness. Moreover, curves L5 to L8 in FIGS. 29 to 32 show the relationship between brightness and position on each image.
[0218] As shown by curve L5 in Fig. 29 and curve L6 in Fig. 30, a normal image corresponding to the field of view of the endoscope 201 is blown out due to reflected light from the retroreflection unit 320 of the treatment tool 301B, and exceeds the dynamic range of the image sensor 2241. Furthermore, as shown by curve L7 in Fig. 31, a normal image when the field of view of the endoscope 201 is poor exceeds the dynamic range of the image sensor 2241 due to cloudiness of the perfusion fluid.
[0219] Therefore, as shown by curve L8 in Figure 32, the HDR image generation unit 2233 generates an HDR image using two corrected images with differences in brightness and a normal image, thereby preventing the image from being blown out even when a retroreflective unit 320 is provided on the treatment tool 301B.
[0220] According to the above-described embodiment 3, the same effects as those of the above-described embodiment 1 are achieved, and even if the field of view of the endoscope 201 deteriorates, treatment of the treatment target area 100 by the treatment tool 301B can be continued.
[0221] (Fourth embodiment) Next, a fourth embodiment will be described. In the fourth embodiment, the movement of the treatment tool is detected and the detection result is displayed. Specifically, the configurations of the image processing unit and the treatment tool are different in the fourth embodiment. The configurations of the image processing unit and the treatment tool according to the fourth embodiment will be described below. Note that the same components as those in the treatment system 1 according to the first embodiment described above are assigned the same reference numerals and detailed description thereof will be omitted.
[0222] [Functional configuration of image processing unit] FIG. 33 is a block diagram illustrating a functional configuration of an image processing device according to the fourth embodiment. 33 further includes a motion detection unit 2234 in addition to the configuration of the image processing unit 222 according to the above-described first embodiment. Furthermore, the image processing unit 222C includes a composite image generation unit 2228C instead of the composite image generation unit 2228.
[0223] The motion detection unit 2234 detects the amount of movement of the treatment tool based on a scale portion provided on the treatment tool that appears in a normal image or a corrected image, which will be described later, and outputs this detection result to the composite image generation unit 2228C. Specifically, the motion detection unit 2234 detects the amount of movement of the treatment tool by detecting the scale portion of the scale portion that appears in the normal image or corrected image that is consecutive in time, using well-known pattern matching or the like. Here, the amount of movement is, for example, the amount of movement of the tip of the ultrasound transducer 312a when the treatment tool performs treatment on the treatment target region 100.
[0224] Based on the detection result of the movement detection unit 2234, the composite image generation unit 2228C generates a composite image by combining the corrected image or the emphasized image with information relating to the movement amount of the treatment tool.
[0225] [Configuration of processing tool] FIG. 34 is a schematic diagram showing a schematic configuration of a part of a treatment tool according to the fourth embodiment. A treatment tool 301C shown in FIG. 34 has, in addition to the configuration of the treatment tool 301 described above, indicator portions 340 formed at predetermined intervals from the ultrasonic transducer 312a toward the proximal end along the longitudinal direction.
[0226] The image processing unit 222C configured as described above executes the same turbidity response control process as in the first and second embodiments described above. In this case, the composite image generation unit 2228C combines information on the amount of movement of the treatment tool 301C with the corrected image or the emphasized image based on the detection result of the motion detection unit 2234. The display image generation unit 2229 generates a display image using the composite image generated by the composite image generation unit 2228C and outputs this display image to the display device 203. As a result, even if the field of view of the endoscope 201 becomes poor due to treatment with the treatment tool 301C, the amount of movement of the treatment tool 301C is displayed, so the surgeon can grasp the amount of movement (travel distance) of the treatment tool 301C, and can perform treatment with the treatment tool 301C without interruption.
[0227] According to the above-described embodiment 4, the same effects as those of the above-described embodiment 1 are achieved, and even if the field of view of the endoscope 201 deteriorates, treatment of the treatment target area 100 by the treatment tool 301C can be continued.
[0228] (Modifications of Embodiments 1 to 4) In the above-described first to fourth embodiments, the display image generation unit 2229 switches the image to be output to the display device 203 in accordance with the mode set in the endoscope control device 202, but the present invention is not limited to this. For example, the display image generation unit 2229 may switch the image to be output to the display device 203 based on a drive signal and a synchronization signal (VT) for the treatment tool 301 input from the treatment tool control device 302. Specifically, when either a drive signal for driving the treatment tool 301 or a synchronization signal (VT) is input from the treatment tool control device 302, the display image generation unit 2229 outputs one or more of a corrected image, an emphasized image, and a composite image to the display device 203.
[0229] This allows the surgeon to switch the content of the display image displayed on the display device 203 without having to change the mode of the endoscope control device 202 each time, allowing the surgeon to perform cutting on the treatment target area 100 using the ultrasonic probe 312 without performing complicated tasks.
[0230] Furthermore, the display image generation unit 2229 switches the type of image to be output to the display device 203 in accordance with the synchronization signal, so that the type of image displayed by the display device 203 switches smoothly, preventing the surgeon from feeling uncomfortable and reducing the burden on the surgeon.
[0231] (Other embodiments) Furthermore, in the first to fourth embodiments of the present disclosure, the treatment for turbidity due to bone powder or the like in a liquid such as a perfusion solution has been described, but the present invention is not limited to a liquid, and can also be applied to an air environment. The first to third embodiments can also be applied to a reduction in visibility in the field of view of an endoscope due to cutting debris, fat mist, etc., generated during an air-based treatment at a joint site.
[0232] Furthermore, in the first to fourth embodiments of the present disclosure, treatments on the knee joint have been described, but the present disclosure can also be applied to other parts (such as the spine) in addition to the knee joint.
[0233] Furthermore, the first to fourth embodiments of the present disclosure can be applied to turbidity other than bone powder, such as debris of soft tissue, synovial membrane, fat, etc., and other noise (cavitation of bubbles, etc.). For example, the first to third embodiments can be applied to turbidity or visual field degradation caused by cuttings of soft tissue such as cartilage, synovial membrane, fat, etc., which are factors that cause visual field degradation due to treatment with the treatment tool 301.
[0234] Furthermore, the first to fourth embodiments of the present disclosure can also be applied to treatment in liquid using the treatment tool 301, which can reduce deterioration of the field of view due to fine bubbles generated by factors such as cavitation caused by ultrasonic vibrations of the treatment tool 301.
[0235] Furthermore, the first to fourth embodiments of the present disclosure can be applied even when the field of view of the endoscope 201 is obstructed by a relatively large piece of tissue. In this case, the endoscope control device 202 may determine whether the field of view of the endoscope 201 is obstructed by an obstruction based on the normal image, and if it is determined that the field of view is obstructed by an obstruction, may perform image processing to remove the obstruction using a well-known technique. In this case, the endoscope control device 202 may perform image processing to a degree that does not affect processing, using the size of the treatment area by the treatment tool 301, the time for which the treatment target site 100 is obstructed, etc.
[0236] Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the treatment systems according to the first to fourth embodiments of the present disclosure. For example, some components may be omitted from all the components described in the treatment systems according to the first to third embodiments of the present disclosure. Furthermore, the components described in the treatment systems according to the first to third embodiments of the present disclosure may be appropriately combined.
[0237] Furthermore, in the treatment systems according to the first to fourth embodiments of the present disclosure, the above-described "unit" can be read as "means" or "circuit," etc. For example, a control unit can be read as control means or a control circuit.
[0238] In addition, the programs to be executed by the treatment systems according to embodiments 1 to 4 of the present disclosure are provided as file data in an installable or executable format stored in a computer-readable storage medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.
[0239] Furthermore, the programs executed by the treatment systems according to the first to fourth embodiments of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.
[0240] In the explanation of the flowcharts in this specification, the order of processing between steps is clearly indicated using expressions such as "first," "then," and "continue," but the order of processing required to implement the present invention is not uniquely determined by these expressions. That is, the order of the processes in the flowcharts described in this specification can be changed as long as no contradictions are present. Furthermore, the program is not limited to such a simple branching process, and branching may be performed by comprehensively determining more determination items.
[0241] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure of the present invention. [Explanation of symbols]
[0242] 1 Treatment System 2 Endoscopic devices 3 Treatment Device 4 Guiding Devices 5 Perfusion device 6,7 Lighting equipment 7 Network Control Device 8 Network Server 201, 201A, 201B Endoscope 202 Endoscope control device 203 Display device 204,2244 Imaging unit 205 Operation input section 211 Insertion part 221 Imaging processing unit 222 Image Processing Unit 223 Turbidity detection unit 224a Image sensor 227,315,326,606 CPU 228,316,327,607,2230 memory 301, 301B, 301C Treatment tools 302 Treatment tool control device 303 Foot Switch 311 Treatment tool body 312 Ultrasound Probe 312a Ultrasonic vibrator 320 Retroreflective part 340 Index section 401 Guide body 601 First lighting control unit 602 Second lighting control unit 603 First lighting device 604 Second lighting device 2221 Image data input unit 2222 Image Generation Unit 2223 Turbidity detection unit 2224 Turbidity determination unit 2225 Feature detection unit 2226 Corrected image generation unit 2226a Turbidity estimation section 2226b Histogram generation unit 2226c Representative luminance calculation unit 2226d Correction coefficient calculation unit 2226e Contrast correction section 2227 Enhanced Image Generation Unit 2228,2228A,2228B,2228C Composite image generation unit 2229 Display image generation unit 2231 Image processing control unit 2232 Position detection unit 2233 HDR Image Generation Unit 2234 Motion detection unit
Claims
1. an image acquisition unit that acquires image data that includes at least a portion of a region of a living body to be treated by the energy treatment device; a feature amount detection unit that detects a region including at least a part of the energy treatment device from an image corresponding to the image data; the image data, and a detection result detected by the feature amount detection unit; an enhanced image generating unit that generates enhanced image data in which the region is enhanced compared to other regions based on the above; a corrected image generation unit that generates corrected image data by performing gradation correction on an image corresponding to the image data; a composite image generating unit that generates composite image data by combining the corrected image data and the emphasized image data; a display image generation unit that generates a display image based on the composite image data; Equipped with Image processing device.
2. 2. The image processing device according to claim 1, The detection unit detecting a marking portion provided on the energy treatment device; Image processing device.
3. 3. The image processing device according to claim 2, The marking portion is A retroreflective member. Image processing device.
4. 3. The image processing device according to claim 2, The marking portion is A scale with markings at predetermined intervals. Image processing device.
5. An image acquisition unit that acquires image data that includes at least a portion of a region of a living body to be treated by an energy treatment device; a feature amount detection unit that detects a region including at least a part of the energy treatment device from an image corresponding to the image data; an enhanced image generating unit that generates enhanced image data in which the region is enhanced compared to other regions, based on the image data and the detection result detected by the feature amount detecting unit; a display image generating unit that generates a display image based on the highlighted image data; a turbidity detection unit that detects turbidity for each of the image data sequentially acquired by the image acquisition unit; a turbidity determination unit that determines whether the turbidity is equal to or greater than a predetermined value for each detection by the turbidity detection unit; Equipped with The display image generation unit If the turbidity determination unit determines that the turbidity is equal to or greater than a predetermined value, a display image based on the highlighted image data is generated, generating a display image based on the image data when the turbidity determination unit determines that the turbidity is not equal to or greater than a predetermined value; Image processing device.
6. 2. The image processing device according to claim 1, The detection unit detecting the region by detecting any one of an edge component, a luminance, and an amount of motion contained in the image; Image processing device.
7. An image acquisition unit that acquires image data that includes at least a portion of a region of a living body to be treated by an energy treatment device; a feature amount detection unit that detects a region including at least a part of the energy treatment device from an image corresponding to the image data; an enhanced image generating unit that generates enhanced image data in which the region is enhanced compared to other regions, based on the image data and the detection result detected by the feature amount detecting unit; a display image generating unit that generates a display image based on the highlighted image data; Equipped with The image acquisition unit acquiring the image data at least one of at the start of treatment of a treatment target site by the energy treatment tool and immediately after the start of treatment based on a drive signal of the energy treatment tool; The display image generation unit generating a display image based on the composite image data; Image processing device.
8. 8. The image processing device according to claim 7, a turbidity detection unit that detects turbidity for each of the image data sequentially acquired by the image acquisition unit; a turbidity determination unit that determines whether the turbidity is equal to or greater than a predetermined value for each detection by the turbidity detection unit; Furthermore, The composite image generation unit The image acquisition unit generating the composite image data by combining the image data acquired by the image acquisition unit immediately before the turbidity determination unit determines that the turbidity is equal to or greater than a predetermined value with the enhanced image data; Image processing device.
9. An image acquisition unit that acquires image data that includes at least a portion of a region of a living body to be treated by an energy treatment device; a feature amount detection unit that detects a region including at least a part of the energy treatment device from an image corresponding to the image data; an enhanced image generating unit that generates enhanced image data in which the region is enhanced compared to other regions, based on the image data and the detection result detected by the feature amount detecting unit; a display image generating unit that generates a display image based on the highlighted image data; Equipped with The emphasized image generating unit performing at least one of edge enhancement processing and gradation correction processing on the region to generate the enhanced image data; Image processing device.
10. 2. The image processing device according to claim 1, The display image generation unit outputting at least one of a display image based on the image data and a display image based on the highlighted image data to a display device; Image processing device.
11. an energy treatment device that can be inserted into a subject and can treat a treatment target site; an endoscope that can be inserted into the subject and that can capture an image of at least a region of the living body that is to be treated by an energy treatment tool and generate image data; an image processing device that performs image processing on the image data and outputs the image data to a display device; Equipped with The image processing device includes: an image acquisition unit that acquires the image data; a feature amount detection unit that detects a region including at least a part of the energy treatment device from an image corresponding to the image data; an enhanced image generating unit that generates enhanced image data in which the region is enhanced compared to other regions, based on the image data and the detection result detected by the feature amount detecting unit; a corrected image generation unit that generates corrected image data by performing gradation correction on an image corresponding to the image data; a composite image generating unit that generates composite image data by combining the corrected image data and the emphasized image data; a display image generation unit that generates a display image based on the composite image data; Equipped with Treatment system.
12. The treatment system of claim 11, The energy treatment device includes: an indicator provided at a distal end portion, the indicator being detectable by emphasizing edge components and brightness of a region included in a part of the energy treatment device compared with other regions in an image corresponding to image data captured by the endoscope; Treatment system.
13. An image processing method executed by an image processing device equipped with a processor having hardware, the processor: acquiring image data including at least a portion of a region to be treated by the energy treatment device; detecting a region including at least a part of the energy treatment device from an image corresponding to the image data; generating enhanced image data in which the region is enhanced compared to other regions based on the image data and the detection result of the region; generating corrected image data by performing gradation correction on an image corresponding to the image data; generating composite image data by combining the corrected image data and the enhanced image data; generating a display image based on the composite image data; To carry out the Image processing methods.
