Image processing device, processing system, learning device, and image processing method

Abstract

Provided are an image processing device, a treatment system, a learning device, and an image processing method that make it possible to continue treatment of a treatment site even when the field of view of an endoscope has deteriorated. This image processing device comprises: an image acquisition unit that acquires cloudy image data including at least a portion of an area where cloudiness has occurred, the area being an area of a living body to be treated using an energy treatment tool; an estimation unit that estimates an object included in the image corresponding to the cloudy image data, using a trained model obtained through machine learning of teaching data in which are associated, with one another, a plurality of pieces of annotation image data obtained by annotating the object included in the plurality of treatment images corresponding to the plurality of pieces of treatment image data in which at least the area of the living body to be treated using the energy treatment tool is imaged, and an identification result in which the object included in each of the plurality of treatment images is identified; and a display image generation unit that generates a display image pertaining to the object on the basis of the cloudy image data acquired by the image acquisition unit and the object estimated by the estimation unit.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, a processing system, a learning device, and an image processing method. [Background technology]

[0002] In arthroscopic surgery, a technique is known in which the inside of the joint is inflated with an irrigating fluid such as saline using an irrigation device to ensure a field of view and then the treatment area is treated (see, for example, Patent Document 1). In this technique, the hammering action of an ultrasonic treatment instrument breaks down the bone, generating bone powder (bone shavings) and cerebrospinal fluid, so the irrigating fluid is used to send the bone powder and cerebrospinal fluid out of the field of view of the endoscope to ensure a field of view of the treatment area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4564595 Summary of the Invention [Problem to be solved by the invention]

[0004] In arthroscopic surgery, when bones are continuously crushed by the hammering action of an ultrasonic treatment instrument, a large amount of bone powder is generated. This bone powder disperses in the irrigation fluid, making the fluid cloudy and obstructing the field of view of the arthroscope used to observe the treatment area, making it difficult to see the treatment area.

[0005] However, in the above-mentioned Patent Document 1, if the field of view of the endoscope observing the treatment area becomes cloudy and deteriorates, the perfusion fluid will expel bone powder from the field of view of the endoscope, and treatment on the treatment area must be stopped and the surgeon must wait until the field of view of the endoscope improves, which extends the treatment time and places a burden on both the surgeon and the patient.

[0006] The present disclosure has been made in consideration of the above, and aims to provide an image processing device, a treatment system, a learning device, and an image processing method that can continue treatment on a treatment area even when the field of view of an endoscope deteriorates. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the image processing device according to the present disclosure includes an image acquisition unit that acquires turbid image data of an area where a living body is to be treated with an energy treatment tool, the turbid image data including at least a part of an area where turbidity has occurred; an estimation unit that estimates the object included in an image corresponding to the turbid image data using a trained model that is machine-learned from teacher data that associates a plurality of annotation image data annotated with an object included in a plurality of treatment images corresponding to each of a plurality of treatment image data images of at least the area where a living body is to be treated with the energy treatment tool, and an identification result that identifies the object included in each of the plurality of treatment images; and a display image generation unit that generates a display image related to the object based on the turbid image data acquired by the image acquisition unit and the object estimated by the estimation unit.

[0008] Further, a treatment system according to the present disclosure includes an energy treatment tool, an imaging device, and an image processing device, wherein the energy treatment tool includes a treatment tool main body portion extending from a base end side to a tip end side along a longitudinal direction, and a processing unit provided on the tip end side of the treatment tool main body and capable of treating a living body, and the imaging device includes a housing main body that is insertable into a subject and extends from a base end side to a tip end side along a longitudinal direction, an illumination unit provided on the housing main body and irradiating illumination light toward at least a region where the living body is to be treated by the energy treatment tool, and an imaging unit provided on the housing main body and generating turbidity image data that includes at least a turbid region in the region where the living body is to be treated by the energy treatment tool, The image processing device includes an image acquisition unit that acquires the turbid image data from the imaging unit, an estimation unit that estimates the object included in the image corresponding to the turbid image data using a trained model that is machine-learned from training data that associates multiple annotation image data annotated with objects included in multiple treatment images corresponding to multiple treatment image data images of at least an area where a living body is treated by the energy treatment tool, and identification results that identify the objects included in each of the multiple treatment images, and a display image generation unit that generates a display image related to the object based on the turbid image data acquired by the image acquisition unit and the object estimated by the estimation unit.

[0009] In addition, the learning device according to the present disclosure has a learning unit that generates a trained model by performing machine learning using training data that uses as input data a plurality of treatment image data images of at least an area where a living body is to be treated with an energy treatment tool and a plurality of annotation image data annotated with objects included in a plurality of treatment images corresponding to each of the plurality of treatment image data, and output data being identification results that identify objects included in images corresponding to image data that at least partially include an area where a living body is to be treated with an energy treatment tool.

[0010] Furthermore, an image processing method according to the present disclosure is an image processing method executed by an image processing device provided with a processor having hardware, in which the processor acquires turbid image data in an area where a living body is to be treated with an energy treatment tool and at least a part of which includes a turbid area, estimates the object included in an image corresponding to the turbid image data using a trained model that has been machine-learned from training data that associates a plurality of annotation image data annotated with an object included in a plurality of treatment images corresponding to a plurality of treatment image data images of at least the area where the living body is to be treated with the energy treatment tool with an identification result that identifies the object included in each of the plurality of treatment images, and generates a display image of the object based on the turbid image data and the estimation result of the object. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to achieve the effect of continuing treatment on the treatment site even when the field of view of the endoscope deteriorates. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a treatment system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a state in which a bone hole is formed by an ultrasonic probe according to an embodiment of the present disclosure. [Figure 3A] FIG. 3A is a schematic diagram showing a schematic configuration of an ultrasound probe according to an embodiment of the present disclosure. [Figure 3B] FIG. 3B is a schematic view of the direction of arrow A in FIG. 3A. [Figure 4] FIG. 4 is a block diagram showing an outline of the functional configuration of the entire treatment system according to an embodiment of the present disclosure. [Figure 5] FIG. 5 is a block diagram showing a detailed functional configuration of an endoscope apparatus according to an embodiment of the present disclosure. [Figure 6A]FIG. 6A is a diagram showing a state in which the field of view of an endoscope according to an embodiment of the present disclosure is good. [Figure 6B] FIG. 6B is a diagram illustrating a state in which the field of view of the endoscope according to the embodiment of the present disclosure is poor. [Figure 7] FIG. 7 is a block diagram illustrating a detailed functional configuration of a processing device according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a block diagram showing a detailed functional configuration of a perfusion device according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a block diagram showing a detailed functional configuration of an illumination device according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram illustrating a general configuration of an illumination device according to an embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating the relationship between the transmission characteristics and wavelength bands of a red filter, a green filter, and a blue filter according to an embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram illustrating the relationship between the transmission characteristics and wavelength bands of an IR transmission filter according to an embodiment of the present disclosure. [Figure 13] FIG. 13 is a block diagram illustrating a detailed functional configuration of an image processing unit according to an embodiment of the present disclosure. [Figure 14] FIG. 14 is a block diagram schematically illustrating exchange of some signals constituting an image processing unit according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a block diagram illustrating a detailed functional configuration of a turbidity correction unit according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a flowchart illustrating an outline of a treatment performed by an operator using a treatment system according to an embodiment of the present disclosure. [Figure 17] FIG. 17 is a diagram illustrating an overview of a process executed by an endoscope control device according to an embodiment of the present disclosure in a cutting procedure. [Figure 18] FIG. 18 is a diagram illustrating an example of a first image generated by a first image generating unit according to an embodiment of the present disclosure. [Figure 19] FIG. 19 is a diagram illustrating an example of a second image generated by a second image generating unit according to an embodiment of the present disclosure. [Figure 20] FIG. 20 is a diagram schematically illustrating an estimation result of an object estimated by an estimation unit according to an embodiment of the present disclosure. [Figure 21] FIG. 21 is a diagram illustrating an example of a display image generated by a display image generating unit according to an embodiment of the present disclosure. [Figure 22] FIG. 22 is a diagram schematically illustrating a method for generating a trained model generated by a learning unit according to an embodiment of the present disclosure. [Figure 23] FIG. 23 is a diagram schematically illustrating a method for generating another trained model generated by a learning unit according to a modified example of an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments for carrying out the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. Furthermore, each drawing referred to in the following description merely shows a schematic representation of the shape, size, and positional relationship to the extent that the contents of the present disclosure can be understood. In other words, the present disclosure is not limited to only the shape, size, and positional relationship exemplified in each drawing. Furthermore, in the following description, the same parts in the drawings will be denoted by the same reference numerals.

[0014] [General configuration of processing system] Fig. 1 is a diagram showing a schematic configuration of a treatment system 1 according to one embodiment. The treatment system 1 shown in Fig. 1 treats biological tissue such as bone by applying ultrasonic vibrations to the biological tissue. Here, treatment refers to, for example, removal or cutting of biological tissue such as bone. Note that Fig. 1 illustrates a treatment system for performing anterior cruciate ligament reconstruction as the treatment system 1.

[0015] The treatment system 1 shown in FIG. 1 includes an endoscope apparatus 2, a treatment apparatus 3, a guiding device 4, a perfusion apparatus 5, and an illumination apparatus 6.

[0016] [Configuration of the endoscope device] First, the configuration of the endoscope device 2 will be described. The endoscope device 2 includes an endoscope 201 , an endoscope control device 202 , and a display device 203 .

[0017] The endoscope 201 has a distal end portion of an insertion section 211 inserted into the joint cavity C1 through a first portal P1 that communicates between the inside of the joint cavity C1 of the knee joint J1 of the subject and the outside of the skin. The endoscope 201 illuminates the inside of the joint cavity C1, takes in illumination light (subject image) reflected within the joint cavity C1, and captures the subject image to generate image data.

[0018] The endoscope control device 202 performs various image processing on image data captured by the endoscope 201, and displays a display image corresponding to the image data after this image processing on the display device 203. The endoscope control device 202 is connected to the endoscope 201 and the display device 203 by wire or wirelessly.

[0019] The display device 203 receives data, image data (display image), audio data, etc. transmitted from each device constituting the treatment system 1 via the endoscope control device 202, and displays, notifies, and outputs a display image according to the received data. The display device 203 is configured using a display panel made of liquid crystal or organic EL (Electro-Luminescence).

[0020] [Configuration of processing device] Next, the configuration of the processing device 3 will be described. The treatment device 3 includes a treatment tool 301 , a treatment tool control device 302 , and a foot switch 303 .

[0021] The treatment tool 301 includes a treatment tool body 311, an ultrasonic cutting section 312 (see FIG. 2 described later), and a sheath 313.

[0022] The treatment tool body 311 is formed in a cylindrical shape. The treatment tool body 311 also houses an ultrasonic transducer 312a (see FIG. 2, which will be described later) that is made up of a bolt-clamped Langevin-type transducer and generates ultrasonic vibrations in response to supplied driving power.

[0023] The treatment tool control device 302 supplies driving power to the ultrasonic transducer 312a in response to an operator's operation of the foot switch 303. The supply of driving power is not limited to the operation of the foot switch 303, and may be performed in response to, for example, an operation of an operation unit (not shown) provided on the treatment tool 301.

[0024] The foot switch 303 is an input interface that is operated by the operator with his / her foot when driving the ultrasonic cutting unit 312 .

[0025] Next, the ultrasonic cutting unit 312 will be described. Fig. 2 is a diagram showing how a bone hole 101 is formed by an ultrasonic cutting unit 312. Fig. 3A is a schematic diagram showing a general configuration of the ultrasonic cutting unit 312. Fig. 3B is a schematic diagram seen in the direction of arrow A in Fig. 3A.

[0026] As shown in FIGS. 2, 3A, and 3B, the ultrasonic cutting unit 312 is made of, for example, a titanium alloy and has a generally cylindrical shape. The proximal end of the ultrasonic cutting unit 312 is connected to an ultrasonic vibrator 312a inside the treatment instrument body 311. The ultrasonic cutting unit 312 transmits ultrasonic vibrations generated by the ultrasonic vibrator 312a from the proximal end to the distal end. Specifically, the ultrasonic vibrations in one embodiment are longitudinal vibrations along the longitudinal direction of the ultrasonic cutting unit 312 (the vertical direction in FIG. 2). The distal end of the ultrasonic cutting unit 312 is provided with the ultrasonic vibrator 312a, as shown in FIG. 2.

[0027] The sheath 313 is formed in a cylindrical shape that is longer and thinner than the treatment instrument body 311, and covers a part of the outer periphery of the ultrasonic cutting section 312 from the treatment instrument body 311 to an arbitrary length.

[0028] The ultrasonic vibrator 312a of the ultrasonic cutting section 312 in the treatment instrument 301 configured in this manner is inserted into the joint cavity C1 while being guided by the guiding device 4 inserted into the joint cavity C1 through a second portal P2 that connects the inside of the joint cavity C1 with the outside of the skin.

[0029] Next, the treatment tool 301 generates ultrasonic vibrations while the ultrasonic vibrator 312a of the ultrasonic cutting section 312 is in contact with the bone treatment target area 100, and the hammering action causes the part of the bone that mechanically collides with the ultrasonic vibrator 312a to be crushed into fine particles (see Figure 2).

[0030] Thereafter, when the operator presses the ultrasonic vibrator 312a of the ultrasonic cutting section 312 of the treatment tool 301 into the treatment target portion 100, the ultrasonic vibrator 312a pulverizes the bone while advancing into the treatment target portion 100. As a result, a bone hole 101 is formed in the treatment target portion 100.

[0031] Further, at the proximal end of the treatment tool body 311, a circuit board 317 on which a posture detection section 314, a CPU (Central Processing Unit) 315, and a memory 316 are mounted is provided (see FIGS. 3A and 3B).

[0032] The attitude detection unit 314 includes a sensor that detects the rotation and movement of the treatment tool 301. The attitude detection unit 314 detects movement in three mutually orthogonal axial directions, including an axis parallel to the longitudinal axis of the ultrasonic cutting unit 312, and rotation around each axis. The treatment tool control device 302 determines that the treatment tool 301 is stationary if the detection result of the attitude detection unit 314 does not change for a certain period of time. The attitude detection unit 314 is composed of, for example, a three-axis angular velocity sensor (gyro sensor) and an acceleration sensor.

[0033] The CPU 315 controls the operation of the attitude detection unit 314 and transmits and receives information to and from the treatment tool control device 302. The CPU 315 reads a program stored in the memory 316 into a working area of ​​the memory and executes it, and controls each component through the execution of the program by the processor, thereby enabling the hardware and software to work together and realizing a functional module that meets a predetermined purpose.

[0034] [Configuration of the guiding device] Next, the configuration of the guiding device 4 will be described. In FIG. 1, the guiding device 4 is inserted into the joint cavity C1 through the second portal P2, and guides the insertion of the tip portion of the ultrasonic cutting unit 312 of the treatment tool 301 into the joint cavity C1.

[0035] The guiding device 4 includes a guide body 401, a handle portion 402, and a drainage portion 403 with a cock.

[0036] The guide body 401 is cylindrical and has a through-hole 401a therein through which the ultrasonic cutting unit 312 is inserted (see FIG. 1). The guide body 401 restricts the movement of the ultrasonic cutting unit 312 inserted into the through-hole 401a in a fixed direction and guides the movement of the ultrasonic cutting unit 312. In one embodiment, the cross-sectional shapes of the outer and inner peripheral surfaces of the guide body 401 perpendicular to the central axis are each approximately circular. Furthermore, the guide body 401 becomes thinner toward the tip. In other words, the tip surface 401b of the guide body 401 is an inclined surface that intersects obliquely with the central axis.

[0037] The cock-equipped drainage section 403 is provided on the outer peripheral surface of the guide main body 401 and has a cylindrical shape that communicates with the inside of the guide main body 401. One end of a drainage tube 505 of the perfusion device 5 is connected to the cock-equipped drainage section 403, forming a flow path that communicates between the guide main body 401 and the drainage tube 505 of the perfusion device 5. This flow path is configured to be openable and closable by operating a cock (not shown) provided in the cock-equipped drainage section 403.

[0038] [Configuration of the perfusion device] Next, the configuration of the perfusion device 5 will be described. In FIG. 1, the perfusion device 5 delivers a perfusion fluid such as sterilized physiological saline into the joint cavity C1 and also discharges the perfusion fluid to the outside of the joint cavity C1.

[0039] The perfusion device 5 includes a liquid source 501, a liquid supply tube 502, a liquid supply pump 503, a drain bottle 504, a drain tube 505, and a drain pump 506 (see FIG. 1).

[0040] Fluid source 501 contains an irrigation fluid therein. Fluid source 501 is connected to fluid supply tube 502. The irrigation fluid is sterilized physiological saline or the like. Fluid source 501 is configured using, for example, a bottle or the like.

[0041] The liquid supply tube 502 has one end connected to the liquid source 501 and the other end connected to the endoscope 201 .

[0042] The fluid supply pump 503 supplies the perfusion fluid from the fluid source 501 to the endoscope 201 through the fluid supply tube 502. The perfusion fluid supplied to the endoscope 201 is supplied into the joint cavity C1 from a fluid supply hole formed at the tip of the insertion section 211.

[0043] The drainage bottle 504 stores the irrigation fluid discharged to the outside of the joint cavity C1. A drainage tube 505 is connected to the drainage bottle 504.

[0044] The drainage tube 505 has one end connected to the guiding device 4 and the other end connected to the drainage bottle 504 .

[0045] The drainage pump 506 drains the perfusion fluid in the joint cavity C1 from the guiding device 4 inserted into the joint cavity C1 through the flow path of the drainage tube 505 into the drainage bottle 504. Although the first embodiment uses the drainage pump 506 for explanation, this is not limiting and a suction device provided in the facility may also be used.

[0046] [Configuration of lighting device] Next, the configuration of the lighting device 6 will be described. 1, the illumination device 6 has two light sources that emit two illumination lights having different wavelength bands. The two illumination lights are, for example, white light, which is visible light, and infrared light, which is invisible light. The illumination lights from the illumination device 6 are propagated to the endoscope 201 via a light guide and emitted from the tip of the endoscope 201.

[0047] [Functional configuration of the entire treatment system] Next, the functional configuration of the entire treatment system will be described. FIG. 4 is a block diagram showing an outline of the functional configuration of the entire treatment system 1. As shown in FIG. In addition to the configuration described above (see FIG. 1), the treatment system 1 shown in FIG. 4 further includes a network control device 7 that controls communication throughout the system, and a network server 8 that stores various data.

[0048] The network control device 7 is communicably connected to the endoscope device 2, treatment device 3, perfusion device 5, illumination device 6, and network server 8. While Fig. 4 illustrates a case where the devices are connected wirelessly, they may also be connected by wire. The detailed functional configurations of the endoscope device 2, treatment device 3, perfusion device 5, and illumination device 6 will be described below.

[0049] The network server 8 is communicably connected to the endoscope device 2, the treatment device 3, the perfusion device 5, the illumination device 6, and the network control device 7. The network server 8 stores various data of each device constituting the treatment system 1. The network server 8 is configured using a processor having hardware such as a CPU, and memories such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

[0050] [Functional configuration of the endoscope device] Next, the functional configuration of the above-mentioned endoscope device 2 will be described. FIG. 5 is a block diagram showing a detailed functional configuration of the endoscope device 2. As shown in FIG. As shown in FIGS. 4 and 5, the endoscope device 2 includes an endoscope control device 202, a display device 203, an imaging unit 204 provided in the endoscope 201, and an operation input unit 205.

[0051] The endoscope control device 202 includes an imaging processing unit 221 (image acquisition unit), an image processing unit 222, a turbidity detection unit 223, an input unit 226, a CPU 227, a memory 228, a wireless communication unit 229, a distance sensor driving circuit 230, a distance data memory 231, and a communication interface 232.

[0052] The imaging processing unit 221 has an imaging element drive control circuit 221a that controls the drive of the imaging element 2241 included in the imaging unit 204 provided in the endoscope 201, and an imaging element signal control circuit 221b that is provided in a patient circuit 202b electrically insulated from the primary circuit 202a and controls the signal of the imaging element 224a. The imaging element drive control circuit 221a is provided in the primary circuit 202a. The imaging element signal control circuit 221b is provided in a patient circuit 202b electrically insulated from the primary circuit 202a.

[0053] The image processing unit 222 performs predetermined image processing on image data (RAW data) input via a bus and outputs the result to the display device 203. The image processing unit 222 is configured using a processor having hardware such as a DSP (Digital Signal Processor) or FPGA (Field-Programmable Gate Array). The image processing unit 222 reads a program stored in a memory 228 into a working area of ​​the memory and executes it, and controls each component through the execution of the program by the processor, thereby enabling the hardware and software to work together and realize a functional module that meets a predetermined purpose. The detailed functional configuration of the image processing unit 222 will be described later.

[0054] The turbidity detection unit 223 detects turbidity of the field of view of the endoscope 201 in the joint cavity C1 based on information about turbidity of the field of view of the endoscope 201. Here, the information about turbidity is, for example, a value obtained from image data generated by the endoscope 201, a physical property value (turbidity) of the perfusion fluid, impedance acquired from the treatment device 3, etc.

[0055] FIG. 6A is a diagram showing a state in which the field of view of the endoscope 201 is good. FIG. 6B is a diagram showing a state in which the field of view of the endoscope 201 is poor. 6A and 6B are diagrams each showing a display image corresponding to image data representing the field of view of the endoscope 201 when the surgeon forms a bone hole in the femoral lateral condyle 900. Of these, FIG. 6B shows a schematic representation of a state in which the field of view of the endoscope 201 is clouded due to bone being crushed into fine particles by the operation of the ultrasonic cutting unit 312. That is, FIG. 6B is an example of a display image corresponding to image data (cloudy image data) captured when the perfusion fluid is clouded and the field of view of the endoscope 201 is clouded. In FIG. 6B, fine bones are represented by dots.

[0056] In FIG. 5, the input unit 226 receives the input of a signal input by the operation input unit 205 and the input of a signal from each device constituting the treatment system 1.

[0057] The CPU 227 comprehensively controls the operation of the endoscope control device 202. The CPU 227 reads out a program stored in the memory 228 into a working area of ​​the memory and executes it, and controls each component part through the execution of the program by the processor, thereby controlling the operation of each part of the endoscope control device 202 through cooperation between hardware and software.

[0058] The memory 228 stores various information necessary for the operation of the endoscope control device 202, various programs executed by the endoscope control device 202, image data captured by the imaging unit 204, etc. The memory 228 is configured using, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a frame memory, etc.

[0059] The wireless communication unit 229 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0060] The distance sensor driving circuit 230 drives a distance sensor (not shown) that measures the distance to a predetermined object in an image captured by the imaging unit 204. In the first embodiment, the distance sensor may be provided in the imaging element 2241. In this case, the imaging element 2241 may be provided with phase difference pixels capable of measuring the distance from the imaging element 2241 to the predetermined object, instead of effective pixels. Of course, a ToF (Time of Flight) sensor or the like may be provided near the tip of the endoscope 201.

[0061] Distance data detected by the distance sensor is stored in the distance data memory 231. The distance data memory 231 is configured using, for example, RAM and ROM.

[0062] The communication interface 232 is an interface for communicating with the image capturing unit 204 .

[0063] Of the above-described components, all except the imaging element signal control circuit 221b are provided in the primary circuit 202a and are interconnected by bus wiring.

[0064] The imaging unit 204 is provided in the endoscope 201. The imaging unit 204 has an imaging element 2241, a CPU 242, and a memory 243.

[0065] Under the control of the CPU 242, the image sensor 2241 generates image data by capturing an image of a subject formed by one or more optical systems (not shown), and outputs the generated image data to the endoscope control device 202. The image sensor 2241 is configured using a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0066] The CPU 242 comprehensively controls the operation of the imaging unit 204. The CPU 242 reads out a program stored in the memory 243 into a working area of ​​the memory and executes it, and controls each component through the execution of the program by the processor, thereby controlling the operation of the imaging unit 204 through cooperation between hardware and software.

[0067] The memory 243 stores various information necessary for the operation of the imaging unit 204, various programs executed by the endoscope 201, image data generated by the imaging unit 204, etc. The memory 243 is configured using RAM, ROM, frame memory, etc.

[0068] The operation input unit 205 is configured using input interfaces such as a mouse, keyboard, touch panel, and microphone, and receives operation inputs of the endoscope device 2 from the operator.

[0069] [Functional configuration of the processing device] Next, the functional configuration of the processing device 3 will be described. FIG. 7 is a block diagram showing a detailed functional configuration of the processing device 3. As shown in FIG. As shown in FIGS. 4 and 7, the treatment device 3 includes a treatment tool 301, a treatment tool control device 302, and an input / output unit 304.

[0070] The treatment tool 301 includes an ultrasonic transducer 312 a , a posture detection unit 314 , a CPU 315 , and a memory 316 .

[0071] The attitude detection unit 314 detects the attitude of the treatment tool 301 and outputs the detection result to the CPU 315. The attitude detection unit 314 is configured using at least one of an acceleration sensor and an angular velocity sensor.

[0072] The CPU 315 comprehensively controls the operation of the treatment tool 301 including the ultrasonic transducer 312a. The CPU 315 reads out a program stored in the memory 316 into a working area of ​​the memory and executes it, and controls each component through the execution of the program by the processor, thereby allowing the hardware and software to work together and realizing a functional module that meets a predetermined purpose.

[0073] The memory 316 stores various information necessary for the operation of the treatment tool 301, various programs executed by the treatment tool 301, and identification information for identifying the type, manufacturing date, performance, etc. of the treatment tool 301.

[0074] The treatment tool control device 302 includes a primary circuit 321, a patient circuit 322, a transformer 323, a first power supply 324, a second power supply 325, a CPU 326, a memory 327, a wireless communication unit 328, a communication interface 329, and an impedance detection unit 330.

[0075] The primary circuit 321 generates power to be supplied to the treatment tool 301. The patient circuit 322 is electrically insulated from the primary circuit 321. The transformer 323 electromagnetically connects the primary circuit 321 and the patient circuit 322. The first power supply 324 is a high-voltage power supply that supplies drive power to the treatment tool 301. The second power supply 325 is a low-voltage power supply that supplies drive power to a control circuit in the treatment tool control device 302.

[0076] The CPU 326 comprehensively controls the operation of the treatment tool control device 302. The CPU 326 reads out a program stored in the memory 327 into a working area of ​​the memory and executes it, and controls each component through the execution of the program by the processor, thereby controlling the operation of each part of the treatment tool control device 302 through cooperation between hardware and software.

[0077] The memory 327 stores various information necessary for the operation of the treatment tool control device 302, various programs executed by the treatment tool control device 302, etc. The memory 327 is configured using RAM, ROM, etc.

[0078] The wireless communication unit 328 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi (registered trademark) and Bluetooth (registered trademark).

[0079] The communication interface 329 is an interface for communicating with the treatment tool 301 .

[0080] The impedance detection unit 330 detects the impedance when the treatment tool 301 is driven and outputs the detection result to the CPU 326. Specifically, the impedance detection unit 330 is electrically connected, for example, between the first power supply 324 and the primary circuit 321, detects the impedance of the treatment tool 301 based on the voltage and current supplied by the first power supply 324, and outputs the detection result to the CPU 326. This impedance changes depending on the degree of turbidity (cloudiness) of the perfusion fluid caused by bone powder resulting from treatment with the treatment tool 301. That is, the impedance detection unit 330 detects the turbidity of the perfusion fluid.

[0081] The input / output unit 304 is configured using input interfaces such as a mouse, keyboard, touch panel, and microphone, and output interfaces such as a monitor and speaker, and outputs operation inputs of the endoscopic device 2 by the surgeon and various information to be notified to the surgeon (see Figure 4).

[0082] [Functional configuration of the perfusion device] Next, the functional configuration of the perfusion device 5 will be described. FIG. 8 is a block diagram showing the detailed functional configuration of the perfusion device 5. As shown in Figures 4 and 8, the perfusion device 5 includes a fluid supply pump 503, a drainage pump 506, a fluid supply control unit 507, a drainage control unit 508, an input unit 509, a CPU 510, a memory 511, a wireless communication unit 512, a communication interface 513, a pump CPU 514, a pump memory 515, and a turbidity detection unit 516.

[0083] The liquid supply control unit 507 includes a first drive control unit 571 , a first drive power generation unit 572 , a first transformer 573 , and a liquid supply pump drive circuit 574 .

[0084] The first drive control unit 571 controls the driving of the first drive power generation unit 572 and the liquid feed pump drive circuit 574 .

[0085] The first drive power generation unit 572 generates drive power for the liquid feed pump 503 and supplies this drive power to the first transformer 573 .

[0086] The first transformer 573 electromagnetically connects the first drive power generation unit 572 and the liquid feed pump drive circuit 574 together.

[0087] In fluid supply control unit 507 configured in this manner, first drive control unit 571, first drive power generation unit 572, and first transformer 573 are provided in primary circuit 5a. In addition, fluid supply pump drive circuit 574 is provided in patient circuit 5b that is electrically insulated from primary circuit 5a.

[0088] The drainage control unit 508 includes a second drive control unit 581, a second drive power generation unit 582, a second transformer 583, and a drainage pump drive circuit 584.

[0089] The second drive control unit 581 controls the driving of the second drive power generation unit 582 and the drainage pump drive circuit 584 .

[0090] The second driving power generating unit 582 generates driving power for the drainage pump 506 and supplies the generated driving power to the second transformer 583 .

[0091] The second transformer 583 electromagnetically connects the second drive power generating unit 582 and the drainage pump drive circuit 584 together.

[0092] In drainage control unit 508 configured in this manner, second drive control unit 581, second drive power generation unit 582, and second transformer 583 are provided in primary circuit 5a. Furthermore, drainage pump drive circuit 584 is provided in patient circuit 5b that is electrically insulated from primary circuit 5a.

[0093] The input unit 509 receives operation inputs (not shown) and signal inputs from the devices that make up the treatment system 1, and outputs the received signals to the CPU 510 and the pump CPU 514.

[0094] The CPU 510 and the pump CPU 514 work together to comprehensively control the operation of the perfusion device 5. The CPU 510 reads a program stored in the memory 511 into a working area of ​​the memory and executes it, and controls each component part through the execution of the program by the processor, thereby controlling the operation of each part of the perfusion device 5 through cooperation between hardware and software.

[0095] The memory 511 stores various information necessary for the operation of the perfusion device 5 and various programs executed by the perfusion device 5. The memory 511 is configured using RAM, ROM, and the like.

[0096] The wireless communication unit 512 is an interface for wireless communication with other devices, and is configured using a communication module capable of, for example, Wi-Fi or Bluetooth.

[0097] The communication interface 513 is an interface for communicating with the liquid feed pump 503 and the endoscope 201 .

[0098] The pump memory 515 stores various information necessary for the operation of the liquid feed pump 503 and the liquid discharge pump 506 and various programs executed by the liquid feed pump 503 and the liquid discharge pump 506 .

[0099] The turbidity detection unit 516 detects the turbidity of the perfusion fluid based on one or more of the physical properties, absorbance, impedance, and resistance of the perfusion fluid flowing in the drainage tube 505, and outputs the detection result to the CPU 510.

[0100] In the perfusion device 5 configured as above, an input unit 509, a CPU 510, a memory 511, a wireless communication unit 512, a communication interface 513, and a turbidity detection unit 516 are provided in the primary circuit 5a. Furthermore, an in-pump CPU 514 and an in-pump memory 515 are provided in the pump 5c. The in-pump CPU 514 and the in-pump memory 515 may be provided in the vicinity of the liquid feed pump 503 or in the vicinity of the drainage pump 506.

[0101] [Functional configuration of lighting device] Next, the functional configuration of the lighting device 6 will be described. FIG. 9 is a block diagram showing a detailed functional configuration of the lighting device 6. As shown in FIG. As shown in Figures 4 and 9, the lighting device 6 includes a first lighting control unit 601, a second lighting control unit 602, a first lighting device 603, a second lighting device 604, an input unit 605, a CPU 606, a memory 607, a wireless communication unit 608, a communication interface 609, a CPU 610 in the lighting circuit, and a memory 630 in the lighting circuit.

[0102] The first illumination control unit 601 has a first drive control unit 611 , a first drive power generation unit 612 , a first controller 613 , and a first drive circuit 614 .

[0103] The first drive control unit 611 controls the driving of the first drive power generation unit 612 , the first controller 613 and the first drive circuit 614 .

[0104] The first drive power generation section 612 generates drive power for the first lighting device 603 under the control of the first drive control section 611 , and outputs this drive power to the first controller 613 .

[0105] The first controller 613 controls the first drive circuit 614 in accordance with the drive power input from the first drive power generation unit 612 , thereby controlling the light output of the first illumination device 603 .

[0106] The first driving circuit 614 drives the first lighting device 603 under the control of the first controller 613 to output illumination light.

[0107] In the first illumination control section 601 configured in this manner, a first drive control section 611, a first drive power generation section 612, and a first controller 613 are provided in a primary circuit 6a. Also, a first drive circuit 614 is provided in a patient circuit 6b that is electrically insulated from the primary circuit 6a.

[0108] The second illumination control unit 602 includes a second drive control unit 621 , a second drive power generation unit 622 , a second controller 623 , and a second drive circuit 624 .

[0109] The second drive control unit 621 controls the driving of the second drive power generation unit 622 , the second controller 623 and the second drive circuit 624 .

[0110] The second drive power generation section 622 generates drive power for the second lighting device 604 under the control of the second drive control section 621 , and outputs this drive power to the second controller 623 .

[0111] The second controller 623 controls the second drive circuit 624 in accordance with the drive power input from the second drive power generation section 622, thereby controlling the light output of the second illumination device 604.

[0112] The second driving circuit 624 drives the second lighting device 604 under the control of the second controller 623 to output illumination light.

[0113] In second illumination control section 602 configured in this manner, second drive control section 621, second drive power generation section 622, and second controller 623 are provided in primary circuit 6a. Second drive circuit 624 is provided in patient circuit 6b that is electrically insulated from primary circuit 6a.

[0114] The first illumination device 603 sequentially irradiates the subject with light of mutually different wavelength bands of visible light (hereinafter simply referred to as "visible light") and light of a wavelength band outside of visible light (hereinafter simply referred to as "invisible light") as first illumination light for illuminating the subject via the endoscope 201. Here, the visible light refers to at least one of light in the blue wavelength band (400 nm to 500 nm), light in the green wavelength band (480 nm to 600 nm), and light in the red wavelength band (570 nm to 680 nm). The invisible light refers to infrared light (800 nm to 2500 nm). The configuration of the first illumination device 603 will be described later.

[0115] The second illumination device 604 may be used as illumination for detecting subject information when configured to irradiate a subject with special light as second illumination light for illuminating the subject via the endoscope 201. Alternatively, the first illumination device 603 may emit light in the visible light wavelength band, and the second illumination device 604 may emit light in the invisible light wavelength band.

[0116] The input unit 605 receives input of signals from each device constituting the treatment system 1, and outputs the received signals to the CPU 606 and the CPU 610 in the lighting circuit.

[0117] The CPU 606 and the lighting circuit CPU 610 work together to comprehensively control the operation of the lighting device 6. The CPU 606 reads a program stored in the memory 607 into a working area of ​​the memory and executes it, and controls each component part through the execution of the program by the processor, whereby the hardware and software work together to control the operation of each part of the lighting device 6.

[0118] The memory 607 stores various information necessary for the operation of the lighting device 6 and various programs executed by the lighting device 6. The memory 607 is configured using RAM, ROM, and the like.

[0119] The wireless communication unit 608 is an interface for wireless communication with other devices, and is configured using a communication module capable of Wi-Fi, Bluetooth, or the like.

[0120] The communication interface 609 is an interface for communicating with the lighting circuit 6c.

[0121] The lighting circuit memory 630 stores various information and programs required for the operation of the first lighting device 603 and the second lighting device 604. The lighting circuit memory 630 is configured using RAM, ROM, and the like.

[0122] In the lighting device 6 configured in this manner, the input unit 605, CPU 606, memory 607, wireless communication unit 608, and communication interface 609 are provided in the primary circuit 6a. The first lighting device 603, the second lighting device 604, the lighting circuit CPU 610, and the lighting circuit memory 61A are provided in the lighting circuit 6c.

[0123] [Configuration of first lighting device] Next, the configuration of the above-mentioned first illumination device will be described. FIG. 10 is a schematic diagram showing a general configuration of the first illumination device 603. As shown in FIG. The first lighting device 603 shown in Figure 10 includes a light source 6031 capable of emitting illumination light, a rotary filter 6032, and an IR transmission filter 6033 that is arranged on the optical path L1 of the illumination light emitted by the light source 6031 and can be moved forward and backward by a drive unit not shown.

[0124] The light source 6031 is configured using a light source such as a halogen lamp. The light source 6031 emits light under the drive of the first drive circuit 614 .

[0125] The rotary filter 6032 has a red filter 6032a that transmits light in the red wavelength band (570 nm to 680 nm), a green filter 6032b that transmits light in the green wavelength band (480 nm to 600 nm), a blue filter 6032c that transmits light in the blue wavelength band (400 nm to 500 nm), and a transparent filter 6032d that transmits light (870 nm to 1080 nm) that has passed through the IR transmission filter 6033. The rotary filter 6032 is rotated by a drive unit (not shown), whereby one of the red filter 6032a, green filter 6032b, blue filter 6032c, and transparent filter 6032d is positioned on the optical path of the white light emitted by the light source 6031.

[0126] The IR transmission filter 6033 is disposed so as to be movable forward and backward by a drive unit (not shown) on the optical path L1 of the illumination light emitted by the light source 6031. The IR transmission filter 6033 transmits infrared light (870 nm to 1080 nm), which is the wavelength band of invisible light contained in the illumination light emitted by the light source 6031.

[0127] [Transmission characteristics of each filter] Next, the transmission characteristics of each filter will be described. FIG. 11 is a diagram showing the relationship between the transmission characteristics and wavelength bands of the red filter 6032a, the green filter 6032b, and the blue filter 6032c. FIG. 12 is a diagram showing the relationship between the transmission characteristics of the IR transmission filter 6033 and the wavelength band. 11 and 12, the horizontal axis represents wavelength and the vertical axis represents transmittance. Also, in FIG. 11, the curve L RR represents the transmission characteristics of the red filter 6032a, and curve L GG represents the transmission characteristics of the green filter 6032b, and curve L BB represents the transmission characteristic of the blue filter 6032c. Furthermore, in FIG. IRR indicates the transmission characteristics of the IR transmission filter 6033.

[0128] As shown in Figures 11 and 12, the rotary filter 6032 rotates under the drive of a drive unit not shown, thereby transmitting light in the red wavelength band, light in the green wavelength band, light in the blue wavelength band, and light in the infrared wavelength band toward the subject.

[0129] [Detailed functional configuration of the image processing unit] Next, the detailed functional configuration of the image processing unit 222 will be described. FIG. 13 is a block diagram showing the detailed functional configuration of the image processing unit 222. As shown in FIG. FIG. 14 is a block diagram showing a schematic diagram of some of the signal exchanges that make up the image processing unit 222. As shown in FIG.

[0130] The image processing unit 222 shown in Figures 13 and 14 has a switching determination unit 2221, an image generation unit 2222, an image correction unit 2223, a learning unit 2224, a learned model memory 2225, an estimation unit 2226, a display image generation unit 2227, a memory 2228, a turbidity detection unit 2229, and a turbidity determination unit 2230.

[0131] The switching determination unit 2221 determines a trained model to be used when the estimation unit 2226 (described later) performs estimation on an image corresponding to image data, based on one or more switching signals of the treatment time t of the treatment tool 301 on the living body input from the outside, the impedance Z which is an electrical characteristic of the treatment tool 301 on the living body detected by the impedance detection unit 330, and the power supply Pw supplied to the treatment tool 301, and outputs this determination result to the estimation unit 2226. The switching determination unit 2221 also outputs the determination result to the learning unit 2224 via the bus.

[0132] The image generation unit 2222 performs predetermined image processing on image data (RAW data) input from the outside to generate a first image corresponding to the first color (RGB) image data or a second image corresponding to the second image data, which is infrared image data. As shown in Fig. 14, the image generation unit 2222 has a first image generation unit 2222a and a second image generation unit 2222b. In one embodiment, the image generation unit 2222 functions as an image acquisition unit that acquires image data.

[0133] When the first illumination device 603 sequentially irradiates light of each wavelength band of red, green, and blue, the first image generation unit 2222a generates a first image by performing predetermined image processing on the three pieces of image data of red, green, and blue generated by the endoscope 201. Here, the predetermined image processing includes, for example, a synthesis process in which the three pieces of image data of red, green, and blue are mixed in a predetermined ratio to generate a white image, a color correction process, a black level correction process, a noise reduction process, a gamma correction process, and the like.

[0134] The second image generation unit 2222b generates a second image by performing predetermined image processing on the second image data generated by the endoscope 201 when the first illumination device 603 sequentially irradiates infrared light. Here, the predetermined image processing includes color correction processing, black level correction processing, noise reduction processing, gamma correction processing, etc.

[0135] The image correcting unit 2223 performs image correction on the first image and the second image generated by the image generating unit 2222, and outputs the image to the display image generating unit 2227 or the learning unit 2224. The image correcting unit 2223 has a muddyness correcting unit 2223a and an edge enhancing unit 2223b.

[0136] The muddyness correction unit 2223a generates first corrected image data by performing gradation correction on the first image generated by the first image generation unit 2222a, and outputs a first corrected image corresponding to this first corrected image data (hereinafter simply referred to as "first corrected image") to the display image generation unit 2227 or the learning unit 2224. Specifically, the muddyness correction unit 2223a performs gradation correction on the first image to remove factors that deteriorate visibility due to muddiness (mud components) contained in the first image, and generates the first corrected image. Details of the muddyness correction unit 2223a will be described later.

[0137] If the contrast of the image generated by the second image generation unit 2222b is low and sufficient contrast is not obtained, the edge enhancement unit 2223b performs well-known edge enhancement processing on the second image to generate second corrected image data, and outputs a second corrected image corresponding to this second corrected image data (hereinafter simply referred to as the "second corrected image") to the display image generation unit 2227 or the learning unit 2224.

[0138] The learning unit 2224 is provided to perform learning using training data in advance of a treatment. The learning unit 2224 is configured to perform learning when a treatment is not to be performed, for example, when learning is to be performed in advance. Therefore, in general, the learning unit 2224 does not perform learning when a treatment is to be performed, and the learning unit 2224 performs learning when a treatment is not to be performed. The learning unit 2224 performs machine learning on training data (a data set for learning or training data) including multiple image data (RAW data) generated by the endoscope 201, a first image, a second image, a first corrected image, a second corrected image, a treatment time t for the living body by the treatment tool 301 input from the outside, an impedance Z detected by the impedance detection unit 330, a supply power Pw supplied to the treatment tool 301, etc., to generate a pre-trained model. Specifically, the learning unit 2224 uses as input data multiple image data (RAW) generated by the endoscope 201, multiple processed image data (first image, second image, first corrected image in which turbidity has been reduced or removed by the turbidity correction unit 2223a, second corrected image in which edge enhancement has been performed by the edge enhancement unit 2223b), and multiple annotation image data annotated with objects included in the multiple second images, the first corrected images, and the second corrected images, and generates a pre-trained model by performing machine learning using training data in which output data is an identification result of identifying objects included in the first image. The learning unit 2224 generates the pre-trained model using a well-known machine learning method. An example of machine learning is deep learning using a neural network, but machine learning based on other methods may also be applied. For example, machine learning statistical models include a simple linear regression model, ridge regression, lasso regression, elastic net regression, random forest regression, rule-fit regression, gradient boosting tree, extra tree, support vector regression, Gaussian process regression, k-nearest neighbor regression, kernel ridge regression, etc.

[0139] Furthermore, the learning unit 2224 may further use training data including, as input parameters, the treatment time t of the treatment tool 301 on the living body input from outside, the impedance Z detected by the impedance detection unit 330, the supply power Pw supplied to the treatment tool 301, etc., to generate learned models for each of the treatment time t, the impedance Z, and the supply power Pw, and store these learned models in the learned model memory 2225.

[0140] Furthermore, the learning unit 2224 may generate a trained model by further using training data including the turbidity (turbidity component) of the first image detected by the turbidity detection unit 2229 as an input parameter, and store this trained model in the trained model memory 2225. Furthermore, the learning unit 2224 may re-train the trained model stored in the trained model memory 2225 by inputting the image data input to the image processing unit 222 as input data.

[0141] The trained model memory 2225 stores a plurality of trained models. Specifically, the trained model memory 2225 stores trained models corresponding to the treatment time t, the impedance Z, and the supply power Pw. The trained model memory 2225 is configured using RAM, ROM, etc.

[0142] The estimation unit 2226 reads out from the learned model memory 2225 a trained model corresponding to the switching signal input from the switching determination unit 2221, estimates an object included in the first image based on the read trained model and at least one of the first image and the second image, and outputs the estimation result to the display image generation unit 2227. Specifically, the estimation unit 2226 uses the switching signal, the first image, and the second image as input parameters, and outputs the object included in the first image as an output parameter to the display image generation unit 2227. Here, the object is the treatment tool 301 in the liquid in which the powder has been dispersed, the powder dispersed in the liquid, the position of the powder, the position of the treatment tool 301 in the first image, the position of the indicator provided on the treatment tool 301, the amount of movement of the indicator of the treatment tool 301, treatment dust generated by the treatment with the treatment tool 301, the shape of the treatment tool 301, etc.

[0143] The display image generation unit 2227 generates display image data based on at least one of the first image and the second image and the object estimated by the estimation unit 2226, and converts a display image corresponding to this display image data (hereinafter simply referred to as "display image") into a predetermined format, for example, converting the RGB format into the YCbCr format, and outputs the converted image to the display device 203. Specifically, the display image generation unit 2227 generates a display image in which position information relating to the position of the area of ​​the object estimated by the estimation unit 2226 is superimposed on the first image.

[0144] The memory 2228 stores various types of information necessary for the operation of the image processing unit 222, various programs executed by the image processing unit 222, various types of image data, etc. The memory 2228 is configured using RAM, ROM, frame memory, etc.

[0145] The turbidity detection unit 2229 detects a change in gradation from at least a partial region of the first image based on the first image generated by the image generation unit 2222, and outputs this detection result to the turbidity determination unit 2230 and the learning unit 2224. Specifically, the turbidity detection unit 2229 detects turbidity in the field of view of the endoscope 201 as at least a partial region of the first image based on the first image generated by the image generation unit 2222. The turbidity detection method by the turbidity detection unit 2229 is the same as the method used for detecting turbidity components by the turbidity estimation unit 2226a of the image correction unit 2223, which will be described later, and therefore detailed descriptions of the detection method will be omitted.

[0146] The turbidity determination unit 2230 determines whether the turbidity detected by the turbidity detection unit 2229 is equal to or greater than a predetermined value, and outputs the determination result to the display image generation unit 2227. Here, the predetermined value is, for example, a value at a level where the treatment site is obscured in the field of view of the endoscope 201 due to turbidity. For example, the value at a level where the treatment site is obscured is a value of high brightness and low saturation (high brightness white).

[0147] [Detailed functional configuration of the turbidity correction unit] Next, the detailed mechanical configuration of the turbidity corrector 2223a will be described. FIG. 15 is a block diagram showing a detailed functional configuration of the turbidity correction unit 2223a. The muddyness correction unit 2223a shown in FIG. 15 includes a muddyness estimation unit 2226a, a histogram generation unit 2226b, a representative brightness calculation unit 2226c, a correction coefficient calculation unit 2226d, and a contrast correction unit 2226e.

[0148] The turbidity estimation unit 2226a estimates the turbidity component for each pixel in the first image. Here, the turbidity component for each pixel refers to the degree of turbidity due to bone powder and debris dissolved in the perfusion fluid, which is a factor in degrading the gradation in the first image. Factors that degrade image quality include phenomena caused by the dissolution of biological tissues such as bone powder, debris, blood, and bone marrow in the perfusion fluid, as well as smoke and sparks during treatment with the treatment tool 301. The following describes the turbidity of the perfusion fluid when bone powder dissolves. Perfusion fluid in which biological tissues have been dissolved is characterized by high brightness, low saturation (low color reproduction), and low contrast.

[0149] For this reason, the turbidity estimation unit 2226a calculates the contrast or brightness and saturation of the first image to estimate the turbidity component of the field of view of the endoscope 201. Specifically, the turbidity estimation unit 2226a estimates the turbidity component H(x, y) based on the R value, G value, and B value of the pixel at the coordinates (x, y) in the first image.

[0150] Here, if the R value, G value, and B value at coordinates (x, y) are Ir, Ig, and Ib, respectively, the turbidity component H(x, y) of the pixel at coordinates (x, y) is estimated by the following equation (1). H(x,y)=min(Ir,Ig,Ib) ···(1)

[0151] The turbidity estimation unit 2226a performs the calculation of the above-mentioned formula (1) for each pixel of the first image. The turbidity estimation unit 2226a sets a scan area F (small area) of a predetermined size for the first image. The size of this scan area F is, for example, a predetermined size of m×n pixels (m and n are natural numbers). In the following description, the pixel at the center of the scan area F will be referred to as the reference pixel. Furthermore, in the following description, each pixel around the reference pixel in the scan area F will be referred to as a neighboring pixel. Furthermore, in the following description, the scan area F will be described as being formed to a size of, for example, 5×5 pixels. Of course, the scan area F can also be applied if it is just one pixel.

[0152] The turbidity estimation unit 2226a calculates (Ir, Ig, Ib) for each pixel in the scan area F while shifting the position of the scan area F relative to the first image, and estimates the minimum value among them as the turbidity component H(x, y) of the reference pixel. Pixel values ​​in high-brightness, low-saturation areas in the first image have similar and large R, G, and B values, so the value of min(Ir, Ig, Ib) is large. That is, areas with high brightness and low saturation have a large value for the turbidity component H(x, y).

[0153] In contrast, pixel values ​​in low brightness or high saturation areas have small R, G, or B values, so the value of min(Ir, Ig, Ib) is small. That is, in low brightness or high saturation areas, the turbidity component H(x, y) has a small value.

[0154] Thus, the turbidity component H(x,y) increases as the concentration of bone powder dissolved in the perfusion fluid increases (as the white color of the bone powder increases), and decreases as the concentration of bone powder dissolved in the perfusion fluid decreases. In other words, the turbidity component H(x,y) increases as the color (white) of the perfusion fluid increases due to the bone powder dissolved in the perfusion fluid, and decreases as the color of the perfusion fluid decreases.

[0155] The turbidity estimation unit 2226a estimates the turbidity component H(x,y) using the above-mentioned formula (1), but is not limited to this, and any index showing high brightness and low saturation can be used as the turbidity component. The turbidity estimation unit 2226a may estimate the turbidity component using one or more of the local contrast value, edge intensity, color density, and object distance. Furthermore, the above-mentioned turbidity detection unit 2229 detects turbidity (turbidity component) using the same method as the turbidity estimation unit 2226a.

[0156] The histogram generation unit 2226b determines the distribution of a histogram in a local region including a reference pixel of the first image and neighboring pixels around the reference pixel, based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a. The degree of change in this turbidity component (x, y) serves as an index for determining the region to which each pixel belongs in the local region. Specifically, the degree of change in this turbidity component (x, y) is determined based on the difference in the turbidity component H(x, y) between the reference pixel and neighboring pixels in the local region.

[0157] That is, the histogram generation unit 2226b generates a brightness histogram for a local region including neighboring pixels for each reference pixel based on the first image input from the first image generation unit 2222a and the turbidity component H(x, y) input from the turbidity estimation unit 2226a. A general histogram is generated by regarding pixel values ​​in the target local region as brightness values ​​and counting the frequency of pixel values ​​one by one.

[0158] In contrast, the histogram generating unit 2226b according to the first embodiment weights the count value for the pixel value of a neighboring pixel according to the turbidity component H(x,y) between the reference pixel and the neighboring pixel in the local region. The count value for the pixel value of the neighboring pixel is, for example, a value in the range of 0.0 to 1.0. The count value is set so that the larger the difference between the turbidity component H(x,y) between the reference pixel and the neighboring pixel is, the smaller the value becomes, and the smaller the difference between the turbidity component H(x,y) between the reference pixel and the neighboring pixel is, the larger the value becomes. Furthermore, the local region is formed to have a size of, for example, 7x7 pixels.

[0159] In general, if a histogram is generated based only on luminance, the luminance of neighboring pixels with large differences from the luminance of the pixel of interest will also be counted. Therefore, it is desirable to generate a local histogram based on the image region to which the pixel of interest belongs.

[0160] In contrast, in generating a brightness histogram in one embodiment, a count value for each pixel in a local region in the first image data is set according to the difference in the turbidity component H(x,y) between a reference pixel and each of the neighboring pixels in the local region in the first image data. Specifically, the count value is calculated using, for example, a Gaussian function so that the larger the difference in the turbidity component H(x,y) between the reference pixel and the neighboring pixels, the smaller the count value, and the smaller the difference in the turbidity component H(x,y) between the reference pixel and the neighboring pixels, the larger the count value (see, for example, Japanese Patent No. 6720012 or Japanese Patent No. 6559229, where the haze component is replaced with the turbidity component).

[0161] The method of calculating the count value by the histogram generation unit 2226b is not limited to a Gaussian function, and any method may be used as long as the count value decreases as the difference between the values ​​of the reference pixel and the neighboring pixels increases. For example, the histogram generation unit 2226b may calculate the count value using a lookup table or a table approximated by a broken line instead of a Gaussian function.

[0162] The histogram generating unit 2226b may also compare the difference in value between the reference pixel and the neighboring pixel with a threshold, and if the difference is equal to or greater than the threshold, reduce the count value of the neighboring pixel (for example, set it to 0.0).

[0163] Furthermore, the histogram generation unit 2226b does not necessarily have to use the frequency of pixel values ​​as count values. For example, the histogram generation unit 2226b may use each of the R value, G value, and B value as count values. Furthermore, the histogram generation unit 2226b may use the G value as a luminance value as count value.

[0164] The representative luminance calculation unit 2226c calculates a representative luminance based on statistical information of the luminance histogram input from the histogram generation unit 2226b. The representative luminance is the luminance of the low luminance part, the luminance of the high luminance part, and the luminance of the intermediate luminance part of the effective luminance range of the luminance histogram. The luminance of the low luminance part is the minimum luminance of the effective luminance range. The luminance of the high luminance part is the maximum luminance of the effective luminance range. The luminance of the intermediate luminance part is the centroid luminance. The minimum luminance is the luminance at which the cumulative frequency is 5% of the maximum value in a cumulative histogram created from the luminance histogram. The maximum luminance is the luminance at which the cumulative frequency is 95% of the maximum value in a cumulative histogram created from the luminance histogram. The centroid luminance is the luminance at which the cumulative frequency is 50% of the maximum value in a cumulative histogram created from the luminance histogram.

[0165] The percentages of cumulative frequency corresponding to the minimum luminance, maximum luminance, and centroid luminance, 5%, 50%, and 95%, can be changed as appropriate. Furthermore, the luminance of the intermediate luminance portion is set to the centroid luminance in the cumulative histogram, but this is not limited thereto, and the centroid luminance does not necessarily have to be calculated from the cumulative frequency. For example, the luminance of the intermediate luminance portion can also be set to the luminance with the highest frequency in the luminance histogram.

[0166] The correction coefficient calculation unit 2226d calculates a correction coefficient for correcting the contrast in a local region based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the statistical information input from the representative luminance calculation unit 2226c. Specifically, when contrast correction is performed by histogram expansion, the correction coefficient calculation unit 2226d calculates a coefficient for histogram expansion using the centroid luminance and maximum luminance from the statistical information.

[0167] Here, histogram expansion is a process that enhances contrast by expanding the effective luminance range of the histogram (see, for example, Japanese Patent No. 6720012 or Japanese Patent No. 6559229). Note that the correction coefficient calculation unit 2226d uses histogram expansion as a means of achieving contrast correction, but is not limited to this. Histogram flattening, for example, may also be applied as a means of achieving contrast correction. For example, the correction coefficient calculation unit 2226d may use a cumulative histogram or a table that approximates a broken line as a method of achieving histogram flattening. This cumulative histogram is obtained by sequentially accumulating the most frequent values ​​of the luminance histogram.

[0168] The contrast correction unit 2226e performs contrast correction of the reference pixel of the first image data for the first image input from the first image generation unit 2222a based on the turbidity component H(x, y) input from the turbidity estimation unit 2226a and the correction coefficient input from the correction coefficient calculation unit 2226d (see, for example, Patent Publication No. 6720012 or Patent Publication No. 6559229).

[0169] The muddy correction unit 2223a configured in this way estimates the muddy component H(x, y) based on the first image, calculates a brightness histogram and a representative brightness using this estimation result, calculates a correction coefficient for correcting contrast within a local region, and performs contrast correction based on the muddy component H(x, y) and the correction coefficient. In this way, the muddy correction unit 2223a can generate a first corrected image in which muddyness has been removed from the first image.

[0170] [Summary of treatment] Next, an outline of the treatment performed by the surgeon using the treatment system 1 will be described. FIG. 16 is a flowchart outlining a procedure performed by an operator using the treatment system 1. As shown in FIG. The operator performing the procedure may be a single doctor, or two or more including a doctor and an assistant.

[0171] As shown in FIG. 16, first, the surgeon forms a first portal P1 and a second portal P2 that respectively connect the inside of the joint cavity C1 of the knee joint J1 with the outside of the skin (step S1).

[0172] Next, the surgeon inserts the endoscope 201 into the joint cavity C1 through the first portal P1, inserts the guiding device 4 into the joint cavity C1 through the second portal P2, and inserts the treatment tool 301 into the joint cavity C1 under the guidance of the guiding device 4 (step S2). Note that, although the case where two portals are formed and then the endoscope 201 and the treatment tool 301 are inserted into the joint cavity C1 through the first portal P1 and the second portal P2 has been described here, it is also possible to form the first portal P1 and insert the endoscope 201 into the joint cavity C1, and then form the second portal P2 and insert the guiding device 4 and the treatment tool 301 into the joint cavity C1.

[0173] Thereafter, the surgeon brings the ultrasonic cutting part 312 into contact with the bone to be treated while visually checking the endoscopic image of the inside of the joint cavity C1 displayed on the display device 203 (step S3).

[0174] Next, the surgeon performs ablation treatment using the treatment tool 301 while viewing the endoscopic image displayed on the display device 203 (step S4). Details of the processing of the treatment system 1 during ablation treatment will be described later.

[0175] Thereafter, the display device 203 performs a display and notification process for displaying information about the inside of the joint cavity C1 and the state after the cutting treatment (step S5). The endoscope control device 202 stops the display and notification after a predetermined time has elapsed after the display and notification process. The surgeon then finishes the treatment using the treatment system 1.

[0176] [Details of cutting procedure] Next, the cutting treatment in step S4 of FIG. 16 will be described in detail. FIG. 17 is a diagram for explaining an outline of the processing executed by the endoscope control device 202 in the cutting treatment. In the following description, each process is executed under the control of the CPU of each control device, but the processes may be executed collectively by any one of the control devices such as the network control device 7, for example.

[0177] The CPU 227 communicates with each device, sets control parameters for each of the treatment device 3 and the perfusion device 5, and inputs the control parameters for each of the treatment device 3 and the perfusion device 5 (step S11).

[0178] Next, the CPU 227 determines whether or not the devices of the parts constituting the treatment system 1 have entered an output ON state (step S12). If the CPU 227 determines that the devices of the parts constituting the treatment system 1 have entered an output ON state (step S12: Yes), the endoscope control device 202 proceeds to step S13, which will be described later. On the other hand, if the CPU 227 determines that the devices of the parts constituting the treatment system 1 have not entered an output ON state (step S12: No), the CPU 227 continues this determination until the devices of the parts constituting the treatment system 1 enter an output ON state.

[0179] Thereafter, the first image generating section 2222a and the second image generating section 2222b acquire image data from the imaging section 204 and generate a first image and a second image (step S13).

[0180] FIG. 18 is a diagram showing an example of the first image generated by the first image generating unit 2222a. FIG. 19 is a diagram showing an example of the second image generated by the second image generating section 2222b. 18 and 19, the first and second images are taken when the field of view of the endoscope 201 is poor. That is, the case of image data (turbid image data) captured when the perfusion fluid is turbid will be described.

[0181] 18, the first image generating unit 2222a generates a first image Q1 based on image data (three image data of red, green, and blue) captured with visible light by the endoscope 201. In this case, the operator cannot grasp the position of the ultrasonic cutting unit 312 from the first image Q1 due to the turbidity of the perfusion fluid.

[0182] 19, the second image generation unit 2222b generates a second image Q2 based on image data captured using invisible infrared light of an area that is the same area as the field of view of the endoscope 201 as the first image Q1 and that includes at least the ultrasonic cutting portion 312. In this case, the surgeon can grasp the outline of the ultrasonic cutting portion 312 from the second image Q2 regardless of the turbidity of the perfusion fluid because the second image generation unit 2222b captures the image using invisible infrared light, but cannot grasp the position or turbidity of the living body, etc., because this differs from the actual situation.

[0183] Next, the turbidity detection unit 2229 detects turbidity in the field of view of the endoscope 201 based on the first image generated by the first image generation unit 2222a (step S14). Specifically, the turbidity detection unit 2229 detects turbidity in the field of view of the endoscope 201 using any one of the luminance, saturation, and contrast of the first image.

[0184] Thereafter, the turbidity determination unit 2230 determines whether the turbidity of the field of view of the endoscope 201 detected by the turbidity detection unit 2229 is equal to or greater than a predetermined value (step S15). The turbidity determination unit 2230 determines whether the turbidity component of the field of view of the endoscope 201 detected by the turbidity detection unit 2229 is equal to or greater than a predetermined value. If the turbidity determination unit 2230 determines that the turbidity component of the field of view of the endoscope 201 detected by the turbidity detection unit 2229 is equal to or greater than a predetermined value (step S15: Yes), the endoscope control device 202 proceeds to step S16, which will be described later. On the other hand, if the turbidity determination unit 2230 determines that the turbidity component of the field of view of the endoscope 201 detected by the turbidity detection unit 2229 is not equal to or greater than the predetermined value (step S15: No), the endoscope control device 202 proceeds to step S21, which will be described later.

[0185] In step S16, the estimation unit 2226 selects a trained model stored in the trained model memory 2225 based on the determination result input from the switching determination unit 2221.

[0186] Next, the estimation unit 2226 estimates the position of the ultrasonic cutting portion 312 from at least a portion of the first image based on the switching signal from the switching determination unit 2221 and at least one of the first image generated by the first image generation unit 2222a and the second image generated by the second image generation unit 2222b (step S17).

[0187] FIG. 20 is a diagram schematically illustrating the estimation result of the object estimated by the estimation unit 2226. As shown in Figure 20, the estimation unit 2226 uses the switching signal and the second image as input data and outputs the estimation result of the position or area G1 of the ultrasonic cutting portion 312 included in the second image Q3 as output data to the display image generation unit 2227.

[0188] Next, the display image generation unit 2227 generates a display image in which guide information for guiding the position of the treatment tool 301 shown in the first image is superimposed on the first image based on the estimation result estimated by the estimation unit 2226, and outputs the display image to the display device 203 (step S18).

[0189] FIG. 21 is a diagram showing an example of a display image generated by the display image generating unit 2227. As shown in FIG. 21, the display image generating unit 2227 generates a display image Q4 in which guide information G2 corresponding to the position or area G1 of the ultrasonic cutting unit 312 is superimposed on a first image Q1. As a result, even if the field of view of the endoscope 201 observing the treatment tool 301 becomes cloudy and deteriorates, the operator can perform cutting of the treatment target site 100 with the ultrasonic cutting unit 312 without interruption, because the guide information G2 displays the position of the ultrasonic cutting unit 312, which is the tip of the treatment tool 301, with a frame that is emphasized compared to other areas.

[0190] In step S19, the CPU 227 determines whether the surgeon is continuing the treatment on the subject. Specifically, the CPU 227 determines whether the treatment instrument control device 302 is supplying power to the treatment instrument 301. If the treatment instrument control device 302 is supplying power to the treatment instrument 301, the CPU 227 determines that the surgeon is continuing the treatment on the subject. If the treatment instrument control device 302 is not supplying power to the treatment instrument 301, the CPU 227 determines that the surgeon is not continuing the treatment on the subject. If the CPU 227 determines that the surgeon is continuing the treatment on the subject (step S19: Yes), the endoscope control device 202 proceeds to step S20, which will be described later. On the other hand, if the CPU 227 determines that the surgeon is not continuing the treatment on the subject (step S19: No), the endoscope control device 202 ends this process.

[0191] In step S20, the CPU 227 determines whether or not the output of each device constituting the treatment system 1 has been turned off. If the CPU 227 determines that the output of each device constituting the treatment system 1 has been turned off (step S19: Yes), the endoscope control device 202 ends this processing. On the other hand, if the CPU 227 determines that the output of each device constituting the treatment system 1 has not been turned off (step S10: No), the endoscope control device 202 returns to step S13 described above.

[0192] In step S21, the CPU 227 performs normal control to output a first image to the endoscope control device 202. Specifically, the CPU 227 outputs the first image (color image) generated by the image processing unit 222 to the display device 203 for display. This allows the surgeon to perform treatment using the treatment tool 301 while viewing the first image displayed on the display device 203. After step S21, the endoscope control device 202 proceeds to step S19.

[0193] [Overview of learning by the learning department] Next, the details of the method for generating a trained model by the training unit 2224 will be described. FIG. 22 is a diagram schematically illustrating a method for generating a trained model generated by the training unit 2224.

[0194] 22, the learning unit 2224 generates a pre-trained model by performing machine learning using a plurality of image data generated by the endoscope device 2 as training data D1. As shown in FIG. 22, the training data is a plurality of treatment image data images obtained by capturing an area where a living body is treated with at least a treatment tool 301, which is an energy treatment tool, and includes treatment images W1 to W2 corresponding to processed image data in a state where the field of view is poor due to bone powder or the like generated by the treatment. n (n=an integer equal to or greater than 2), and a plurality of corrected images K1 to K obtained by removing turbidity from the processed images W1 to Wn by the image corrector 2223 described above. m (m=an integer equal to or greater than 2), and a plurality of corrected images K1 to K2 on which annotations or tags for the positions of the areas where the living body is treated by the treatment tool 301 and image processing parameters for the turbidity correction processing have been applied. m The above explanation is based on the correspondence between the treatment images W1 to W n and corrected images K1 to K m When both are used, the treatment images W1 to W n , or corrected images K1 to K m It is also possible to use only one of the above.

[0195] The learning unit 2224 performs machine learning on the training data D1, and generates a trained model for the input image data, which outputs the position G1 (coordinate address) of the area in the image Q4 corresponding to the image data where the living body is treated by the treatment tool 301, which is the target object, as output data of the identification result, and records this trained model in the trained model memory 2225.

[0196] According to the embodiment described above, the display image generation unit 2227 generates and outputs the display image Q3 based on the object included in the first image estimated by the estimation unit 2226, so that even if the field of view of the endoscope 201 deteriorates, treatment of the treatment target area 100 using the treatment tool 301 can be continued.

[0197] Furthermore, according to one embodiment, the display image generation unit 2227 generates and outputs the display image Q3 based on the estimation result of the object included in either the first image or the second image estimated by the estimation unit 2226. As a result, the surgeon can easily confirm the position of the ultrasonic cutting unit 312, and can perform cutting of the treatment target area 100 by the ultrasonic cutting unit 312 without interruption.

[0198] In one embodiment, the display image generation unit 2227 generates a display image in which guide information for guiding the position of the treatment tool 301 included in the first image is superimposed on the first image based on the estimation result estimated by the estimation unit 2226, and outputs the display image to the display device 203. However, this is not limited to this. For example, the image correction unit 2223 may generate a display image using a first corrected image in which turbidity (bone powder) of the first image is corrected based on the estimation result estimated by the estimation unit 2226, and output the display image to the display device 203.

[0199] In addition, in one embodiment, the display image generation unit 2227 generates and outputs the display image Q3 based on the estimation result of the object included in either the first image or the second image estimated by the estimation unit 2226, but it is also possible to generate a display image in which guide information for guiding the position of the treatment tool 301 included in the first image is superimposed on the first corrected image corrected by the turbidity correction unit 2223a, and output the display image to the display device 203.

[0200] In addition, in one embodiment, the estimation unit 2226 estimates the object contained in the second image using a trained model, but this is not limited to this, and the object contained in each of the first image, the first corrected image, and the second corrected image may be estimated.

[0201] (Variation) FIG. 23 is a diagram schematically illustrating a method for generating another trained model generated by the training unit 2224 according to a modified example of an embodiment.

[0202] As shown in FIG. 23, the learning unit 2224 acquires, as the teacher data D2, a plurality of treatment image data obtained by capturing an image of a region where a living body is treated by at least a treatment tool 301 that is an energy treatment tool, and the treatment images U1 to U2 include an indicator portion 320 provided on the treatment tool 301. l (l=an integer equal to or greater than 2), and a plurality of corrected images K1 to K obtained by removing turbidity using the image corrector 2223. m (m=an integer equal to or greater than 2), and a plurality of corrected images O1 to O2 to which annotations or tags for the indicator portion 320 of the treatment tool 301 and image processing parameters for the turbidity correction processing have been applied. m and generate a trained model that outputs, as output data, guide information G1 that guides the position of the area included in the ultrasonic cutting unit 312 according to the position of the indicator 320 provided on the treatment tool 301 included in image Q4. Of course, the learning unit 2224 may perform machine learning on the teacher data D2 to generate a trained model that outputs, as output data, the amount of movement of the indicator 320 provided on the treatment tool 301 included in image Q4. In this case, the estimation unit 2226 estimates the position or amount of movement of the indicator as an object in the first image using the trained model generated by the learning unit 2224 using the teacher data D2, and outputs the estimation result to the image correction unit 2223 and the display image generation unit 2227.

[0203] According to the modification of the embodiment described above, the same effects as those of the above-described embodiment can be achieved, and an image in which the position or movement of the treatment tool 301 is identified can be output.

[0204] In a modified example of the embodiment, the learning unit 2224 may perform machine learning using a plurality of first images and a plurality of second images as training data to generate a trained model that outputs, as output data, correction parameters for color information for correcting an infrared second image into a color image. In this case, the estimation unit 2226 estimates correction parameters for color information in the second image using the trained model generated by the learning unit 2224 using training data consisting of a plurality of first images and a plurality of second images, and outputs the estimation result to the image correction unit 2223 and the display image generation unit 2227. At this time, the image correction unit 2223 corrects the infrared (monochrome) second image into a color image based on the correction parameters for color information in the estimation result estimated by the estimation unit 2226, and outputs the color image to the display image generation unit 2227. In addition to color information, the estimation unit 2226 may also estimate parameters for correcting luminance information of the first image based on luminance information of the second image. As a result, even if the first image is cloudy, the second image can display a color image that reproduces the color of the field of view of the endoscope 201. As a result, the surgeon can easily confirm the position of the ultrasonic cutting unit 312, and can perform cutting of the treatment target area 100 with the ultrasonic cutting unit 312 without interruption.

[0205] (Other embodiments) Furthermore, in one embodiment of the present disclosure, a treatment for turbidity due to bone powder or the like in a liquid such as a perfusion solution has been described, but the present disclosure is not limited to a liquid, and can be applied to an air environment as well. Embodiments 1 to 3 can also be applied to a reduction in visibility in the field of view of an endoscope due to cutting debris, fat mist, etc., generated during an air-based treatment at a joint site.

[0206] Furthermore, in one embodiment of the present disclosure, treatment on the knee joint has been described, but the present disclosure can also be applied to other parts (such as the spine) in addition to the knee joint.

[0207] Furthermore, one embodiment of the present disclosure can be applied to turbidity other than bone powder, for example, debris such as soft tissue, synovial membrane, and fat, and other noise (cavitation such as bubbles). For example, in the first to third embodiments, the present disclosure can be applied to turbidity or visual field degradation caused by cuttings of soft tissue such as cartilage, synovial membrane, and fat, which are factors that cause visual field degradation due to treatment with the treatment tool 301.

[0208] In addition, one embodiment of the present disclosure can also be applied to treatment in liquid using the treatment tool 301, where deterioration of the field of view is caused by fine bubbles generated by factors such as cavitation due to ultrasonic vibrations of the treatment tool 301.

[0209] Furthermore, one embodiment of the present disclosure can be applied even when the field of view of the endoscope 201 is obstructed by a relatively large piece of tissue. In this case, the endoscope control device 202 may determine whether the field of view of the endoscope 201 is obstructed by an obstruction based on the first image, and if it determines that the field of view of the endoscope 201 is obstructed by an obstruction, perform image processing to remove the obstruction using well-known technology. In this case, the endoscope control device 202 may perform image processing to an extent that does not affect processing, using the size of the treatment area by the treatment tool 301, the time the treatment target site 100 is obstructed, etc.

[0210] Furthermore, an embodiment of the present disclosure can also be applied to a case where a filter that can transmit near-infrared light (700 nm to 2500 nm) or an LED that can irradiate near-infrared light is used instead of infrared light.

[0211] Furthermore, in one embodiment of the present disclosure, the learning unit 2224 performs machine learning using training data with multiple image data (multiple processed image data) as input parameters, but it may also be configured to learn to estimate subsequent scenes based on, for example, a scene change.

[0212] Furthermore, in one embodiment of the present disclosure, the output of the estimation unit 2226 may be configured to output data in a format and content that is easy to use in an external device, such as data for reconstructing an image, data including notification information, codec data, etc., regardless of whether correction is required.

[0213] In addition, in one embodiment of the present disclosure, the training data uses image data that includes turbidity due to bone powder during the cutting procedure, but images that include various turbidities that occur during the cutting process, such as mist, blood, bone marrow fluid, and fat chips, can also be used.

[0214] Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the treatment system according to one embodiment of the present disclosure. For example, some components may be omitted from all the components described in the treatment systems according to the first to third embodiments of the present disclosure. Furthermore, the components described in the treatment systems according to the first to third embodiments of the present disclosure may be appropriately combined.

[0215] Furthermore, in the treatment system according to an embodiment of the present disclosure, the above-described "unit" can be read as "means" or "circuit," etc. For example, the control unit can be read as control means or a control circuit.

[0216] In addition, the program to be executed by the treatment system according to one embodiment of the present disclosure is provided as file data in an installable or executable format stored in a computer-readable storage medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, or a flash memory.

[0217] Furthermore, the program executed by the treatment system according to one embodiment of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0218] In the description of the flowcharts in this specification, expressions such as "first," "then," and "continue" are used to clearly indicate the order of processing between steps, but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed within a consistent range. Furthermore, programs are not limited to those consisting of simple branching processing, and branching can be achieved by comprehensively determining more judgment items.

[0219] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that have undergone various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure of the present invention. [Explanation of symbols]

[0220] 1 Treatment System 2 Endoscopic devices 3 Treatment Device 4 Guiding Devices 5 Perfusion device 6,7 Lighting equipment 7 Network Control Device 8 Network Server 201, 201A, 201B Endoscope 202 Endoscope control device 203 Display device 204,2244 Imaging unit 205 Operation input section 211 Insertion part 221 Imaging processing unit 222 Image Processing Unit 223 Turbidity detection unit 224a Image sensor 227,315,326,606 CPU 228,316,327,607,2231 memory 301 Treatment tools 302 Treatment tool control device 303 Foot Switch 311 Treatment tool body 312 Ultrasonic cutting section 312a Ultrasonic vibrator 401 Guide body 601 First lighting control unit 602 Second lighting control unit 603 First lighting device 604 Second lighting device 6031 Light source 6032 Rotating Filter 6032a Red filter 6032b Green filter 6032c Blue filter 6032d Transparent Filter 6033 IR transmission filter 2221 Switching decision unit 2222 Image Generation Unit 2222a 1st image generation section 2222b Second image generation section 2223 Image Correction Unit 2223a Turbidity correction unit 2223b Edge enhancement section 2224 Learning Department 2225 trained model memory 2226 Estimation Department 2226a Turbidity estimation section 2226b Histogram generation unit 2226c Representative luminance part 2226d Correction coefficient calculation unit 2226e Contrast correction section 2227 Display image generation unit 2228 memory 2229 Turbidity detection unit 2230 Turbidity determination unit 2241 Image sensor

Claims

1. an estimation unit that estimates an object from input turbid image data using a trained model that has been machine-learned from training data that associates annotation image data, in which annotations have been applied to an object in turbid image data when treating a living body with an energy treatment tool, with an identification result that identifies the object in the turbid image data; a display image generating unit that generates a display image relating to the object based on the input turbidity image data and the object estimated by the estimation unit; Equipped with The object is an indicator portion provided on the energy treatment device, The estimation unit Estimating the position of the indicator portion in the liquid in which the powder has been dispersed. Image processing device.

2. An estimation unit that estimates an object from input turbid image data using a trained model that has been machine-learned from training data that associates annotation image data, in which annotations have been applied, with an identification result that identifies the object in the turbid image data when treating a living body with an energy treatment tool; a display image generating unit that generates a display image relating to the object based on the input turbidity image data and the object estimated by the estimation unit; Equipped with The object is an indicator portion provided on the energy treatment device, The estimation unit estimating the amount of movement of the indicator portion; Image processing device.

3. An estimation unit that estimates the object from input turbid image data using a trained model that has been machine-learned from training data that associates annotation image data, which has been annotated for the object in turbid image data when treating a living body with an energy treatment tool, with an identification result that identifies the object in the turbid image data; a display image generating unit that generates a display image relating to the object based on the input turbidity image data and the object estimated by the estimation unit; a trained model memory that records a plurality of trained models corresponding to a drive time of the energy treatment device, an electrical characteristic of the living body by the energy treatment device, and a power supply to the energy treatment device; Equipped with The estimation unit selecting one of the plurality of trained models recorded in the trained model memory based on one or more of the driving time, the electrical characteristics, and the supplied power that are externally input; Image processing device.

4. The image processing device according to any one of claims 1 to 3, an image correction unit that generates corrected image data by correcting the input turbidity image data based on the object estimated by the estimation unit; The display image generation unit generating the display image based on either the input turbidity image data or the corrected image data and the object estimated by the estimation unit; Image processing device.

5. The image processing device according to any one of claims 1 to 3, The estimation unit Estimating the position or shape of the object in the liquid in which the powder has been dispersed; Image processing device.

6. The image processing device according to any one of claims 1 to 3, The object is It is a powder dispersed in a liquid, The estimation unit Estimating the position of the powder; Image processing device.

7. 5. The image processing device according to claim 4, The image correction unit acquiring infrared image data from an imaging element capable of receiving invisible light including at least an infrared wavelength band; Image processing device.

8. An energy treatment device, an imaging device, and an image processing device, Equipped with The energy treatment device includes: a treatment instrument main body portion extending from a base end side to a tip end side along a longitudinal direction; a processing unit provided at the distal end side of the treatment instrument body and capable of treating a living body; Equipped with The imaging device is a housing main body that can be inserted into a subject and extends from a base end side to a tip end side along a longitudinal direction; an illumination unit provided in the housing body and configured to irradiate illumination light toward at least an area of ​​a living body to be treated by the energy treatment device; an imaging unit provided in the housing body, the imaging unit generating turbidity image data including at least a part of a turbid region in a region where a living body is treated by the energy treatment device; Equipped with The image processing device includes: an estimation unit that estimates an object from input turbid image data using a trained model that has been machine-learned from training data that associates annotation image data, in which annotations have been applied to an object in turbid image data when treating a living body with an energy treatment tool, with an identification result that identifies the object in the turbid image data; a display image generating unit that generates a display image relating to the object based on the input turbidity image data and the object estimated by the estimation unit; Equipped with The object is an indicator portion provided on the energy treatment device, The estimation unit estimating the amount of movement of the indicator portion; Treatment system.

9. a learning unit that generates a trained model by machine learning using training data in which input data include a plurality of treatment image data images of a region of a living body to be treated by an energy treatment tool and a plurality of annotation image data in which annotations of objects included in a plurality of treatment images corresponding to the plurality of treatment image data are made, and output data include an identification result of identifying objects included in an image corresponding to image data that includes at least a portion of the region of a living body to be treated by an energy treatment tool; The learning unit As the input data, any one or more of a driving time of the energy treatment device, an electrical characteristic of the energy treatment device to the living body, and a supply power to the energy treatment device are further input data, generating the trained models for the driving time, the electrical characteristics, and the supplied power; Learning device.

10. A learning unit is provided which generates a trained model by machine learning using training data in which input data are a plurality of treatment image data images of an area where a living body is to be treated by an energy treatment tool and a plurality of annotation image data annotated with objects contained in a plurality of treatment images corresponding to each of the plurality of treatment image data, and output data are identification results that identify objects contained in images corresponding to image data that at least partially include an area where a living body is to be treated by an energy treatment tool; The treatment image data includes: Image data obtained by imaging a liquid into which powder has been dispersed by treatment of a living body with the energy treatment device, The annotation may be: the position of an indicator portion provided on the energy treatment device included in an image corresponding to the image data; Learning device.

11. A learning unit is provided which generates a trained model by machine learning using training data in which input data are a plurality of treatment image data images of an area where a living body is to be treated by an energy treatment tool and a plurality of annotation image data annotated with objects contained in a plurality of treatment images corresponding to each of the plurality of treatment image data, and output data are identification results that identify objects contained in images corresponding to image data that at least partially include an area where a living body is to be treated by an energy treatment tool, The treatment image data includes: Image data obtained by imaging a liquid into which powder has been dispersed by treatment of a living body with the energy treatment device, The annotation may be: is a movement amount of an indicator portion provided in the energy treatment device included in an image corresponding to the image data. Learning device.

12. The learning device according to any one of claims 9 to 11, The annotation image data includes: Corrected image data obtained by performing a turbidity correction process on each of the processed image data, and the corrected image data is the annotation-applied corrected image data. Learning device.

13. The learning device according to any one of claims 9 to 11, The annotation image data includes: The infrared image data is infrared image data obtained by an imaging element capable of receiving invisible light including at least an infrared wavelength band, and the infrared image data is annotated. Learning device.

14. The learning device according to claim 9, The treatment image data includes: Image data obtained by imaging a liquid into which powder has been dispersed by treating a living body with the energy treatment device. Learning device.

15. An image processing method executed by an image processing device equipped with a processor having hardware, the processor: The object is estimated from the input turbid image data using a trained model obtained by machine learning training data that associates annotation image data, in which annotations have been applied to the object in the turbid image data when treating a living body with an energy treatment tool, with an identification result that identifies the object in the turbid image data; generating a display image relating to the object based on the input turbidity image data and the estimation result of the object; The object is an indicator portion provided on the energy treatment device, The processor: estimating the amount of movement of the indicator portion; To carry out the Image processing methods.

Images