Medical image processing device, operation method of medical image processing device, endoscope system, and medical image processing program

The medical image processing device improves endoscope systems by recognizing regions of interest and adjusting display based on instrument state, enhancing visibility during non-treatment states and reducing interference during treatment, thus aiding medical procedures.

JP7594015B2Active Publication Date: 2024-12-03FUJIFILM CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022545530
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-08-24
Filing Date
2021-07-16
Publication Date
2024-12-03
Estimated Expiration
2041-07-16

AI Technical Summary

Technical Problem

Existing endoscope systems struggle to effectively discriminate and display regions of interest during medical procedures, particularly when instruments are being used, leading to potential interference with the physician's ability to perform tasks like biopsies.

Method used

A medical image processing device that recognizes regions of interest and instrument information, adjusting the display to enhance discrimination power during non-treatment states and reduce it during treatment states, using neural networks to determine the state of instruments and superimpose appropriate symbols or changes in image properties.

Benefits of technology

Enhances the visibility of regions of interest during non-treatment states while minimizing interference during treatment, improving the physician's ability to perform procedures like biopsies by maintaining clear discrimination where necessary.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594015000001
    Figure 0007594015000001
  • Figure 0007594015000002
    Figure 0007594015000002
  • Figure 0007594015000003
    Figure 0007594015000003
Patent Text Reader

Abstract

The purpose of the present invention is to provide a medical image processing apparatus, an endoscope system, a medical image processing method, and a medical image processing program that are capable of displaying a region of interest with appropriate identifiability. The medical image processing apparatus according to an embodiment of the present invention comprises a processor, wherein the processor performs an image acquisition process of acquiring an observation image of a subject, a region of interest recognition process of recognizing a region of interest from the observation image, an instrument information recognition process of recognizing instrument information, which is information on an instrument used to treat the subject, from the observation image, and a display control process of identifiably displaying the observation image on a display device in a manner such that the region of interest has identifiability corresponding to the recognition result of the instrument information.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a medical image processing apparatus, a medical image processing method, an endoscope system, and a medical image processing program. [Background technology]

[0002] The endoscopic system described in Patent Document 1 includes an image acquisition unit that acquires an image of a subject, a recognition unit that performs recognition processing to recognize the subject using the image, a discrimination unit that discriminates operations performed on the subject, a setting unit that sets the recognition unit to be enabled or disabled using the discrimination result of the discrimination unit, and a notification unit that notifies the user of the enabled or disabled state of the recognition unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO2020 / 036224 publication Summary of the Invention

[0004] One embodiment of the present invention provides a medical image processing device, a medical image processing method, an endoscope system, and a medical image processing program capable of displaying a region of interest with appropriate discrimination. [Means for solving the problem]

[0005] A medical image processing device according to a first aspect of the present invention is a medical image processing device equipped with a processor, which performs image acquisition processing to acquire an observation image of a subject, region of interest recognition processing to recognize a region of interest from the observation image, instrument information recognition processing to recognize instrument information, which is information about an instrument used to treat the subject, from the observation image, and display control processing to cause a display device to display the observation image in a manner such that the region of interest has a discrimination ability corresponding to the recognition result of the instrument information.

[0006] In the first aspect of the medical image processing device of the second aspect, in an instrument information recognition process, a processor determines whether the state is a treatment state in which an instrument is performing treatment on a region of interest, a pre-treatment state in which treatment is being prepared, or a non-treatment state which is a state other than the treatment state and the pre-treatment state, based on the instrument information, and in a display control process, displays the observation image with lower discrimination power in the treatment state and the pre-treatment state than in the non-treatment state.

[0007] In the medical image processing device according to the third aspect, in the display control process, the processor displays the observation image with lower discrimination power in the treatment state than in the pre-treatment state.

[0008] In the medical image processing device of the fourth aspect, in the second or third aspect, the processor, in an instrument information recognition process, determines whether the state is treatment, pre-treatment, or non-treatment based on instrument information including at least one of the following: whether an instrument is inserted, the type of inserted instrument, the length of insertion, the operation state of the instrument, the distance between the instrument and the area of ​​interest, and whether the instrument and the area of ​​interest overlap in the observation image.

[0009] In a medical image processing device according to a fifth aspect, which is any one of the second to fourth aspects, the processor, in the display control processing, superimposes a frame surrounding the region of interest on the observation image in the treatment state and / or pre-treatment state.

[0010] In a medical image processing device according to a sixth aspect, which is any one of the second to fifth aspects, the processor, in the display control processing, superimposes a symbol indicating a region of interest on the observation image in the treatment state and pre-treatment state.

[0011] In a medical image processing device according to a seventh aspect, in any one of the second to sixth aspects, the processor, in a display control process, causes at least one of letters, figures, and symbols to be superimposed on a portion of the region of interest in the observation image in the treatment state and the pre-treatment state.

[0012] In a medical image processing device according to an eighth aspect, which is any one of the fifth to seventh aspects, the processor reduces the discriminability of the superimposed display in the treatment state and the pre-treatment state compared to the non-treatment state.

[0013] In a medical image processing device according to a ninth aspect, in any one of the second to eighth aspects, the processor, in a display control process, displays an observation image by changing the color and / or brightness of the area of ​​interest in the treatment state and pre-treatment state compared to the non-treatment state.

[0014] An endoscopic system according to a tenth aspect comprises a medical image processing device according to any one of the first to ninth aspects, a display device for displaying an observation image, and an endoscopic scope to be inserted into a subject, the endoscopic scope having an imaging unit for capturing the observation image.

[0015] A medical image processing method according to an eleventh aspect of the present invention causes a computer to perform an image acquiring step of acquiring an observation image of a subject, a region of interest recognizing step of recognizing a region of interest from the observation image, an instrument information recognizing step of recognizing instrument information, which is information on an instrument used to treat the subject, from the observation image, and a display control step of displaying the observation image on a display device in a manner in which the region of interest has a recognizability according to the recognition result of the instrument information. The medical image processing methods according to the eleventh aspect and each of the following aspects can also be understood as a method of operating a medical image processing device.

[0016] In the medical image processing method of the 12th aspect, in the instrument information recognition step, it is determined based on the instrument information whether the state is a treatment state in which an instrument is performing treatment on the area of ​​interest, a pre-treatment state in which preparation for treatment is being made, or a non-treatment state which is a state other than the treatment state and the pre-treatment state, and in the display control step, the observation image is displayed on the display device with reduced discrimination power in the treatment state and the pre-treatment state compared to the non-treatment state.

[0017] A medical image processing method according to a thirteenth aspect is the twelfth aspect, wherein in the display control step, the observation image is displayed with a lower discrimination power in the treatment state than in the pre-treatment state.

[0018] In the medical image processing method of the 14th aspect, in the 12th or 13th aspect, the instrument information recognition process determines whether the state is treatment, pre-treatment, or non-treatment based on instrument information including at least one of the following: whether an instrument is inserted, the type of inserted instrument, the length of insertion, the operation state of the instrument, the distance between the instrument and the area of ​​interest, and whether the instrument and the area of ​​interest overlap in the observation image.

[0019] A medical image processing program according to the fifteenth aspect causes a computer to execute the medical image processing method according to any one of the eleventh to fourteenth aspects. A non-transitory recording medium having recorded thereon a computer-readable code of the medical image processing program according to the fifteenth aspect can also be cited as one aspect of the present invention. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is an external view of an endoscope system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a main part of the endoscope system. [Diagram 3] FIG. 3 is a functional block diagram of the image processing unit. [Figure 4] FIG. 4 is a diagram showing the configuration of a convolutional neural network. [Diagram 5] FIG. 5 is a diagram showing the convolution process by a filter. [Figure 6] FIG. 6 is a flowchart showing the procedure of the medical image processing method according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a screen for setting definitions of treatment states and the like. [Figure 8] FIG. 8 is a diagram showing an example of a screen for setting the mode of identification display. [Figure 9]FIG. 9 is a diagram showing an example of identification display of a biopsy target range (region of interest). [Figure 10] FIG. 10 is a diagram showing an example of an observation image displayed with the discrimination power of the region of interest reduced. [Figure 11] FIG. 11 is a diagram showing an example of an identification display according to the distance between the tool and the region of interest. [Figure 12] FIG. 12 is a diagram showing an example of an identification display according to the operating state of the tool. [Figure 13] FIG. 13 is a diagram showing another example of an identification display according to the operating state of the tool. [Figure 14] FIG. 14 is a diagram showing still another example of an identification display according to the operating state of the tool. [Figure 15] FIG. 15 is a diagram showing still another example of an identification display according to the operating state of the tool. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a medical image processing apparatus, a medical image processing method, an endoscope system, and a medical image processing program according to the present invention will be described in detail with reference to the accompanying drawings.

[0022] <First embodiment> <Endoscope system configuration> Fig. 1 is an external view of an endoscope system 10 (endoscope system), and Fig. 2 is a block diagram showing the configuration of the main parts of the endoscope system 10. As shown in Figs. 1 and 2, the endoscope system 10 is composed of an endoscope scope 100 (image acquisition unit, endoscope scope), a medical image processing device 200 (medical image processing device, processor, medical image acquisition unit, region of interest recognition unit, instrument information recognition unit, display control unit, recording control unit), a light source device 300 (light source device), and a monitor 400 (display device, display).

[0023] <Configuration of an endoscope> The endoscope 100 includes a handheld operation unit 102 and an insertion section 104 connected to the handheld operation unit 102. An operator (user) holds and operates the handheld operation unit 102, and inserts the insertion section 104 into the body of a subject (living body) for observation. The handheld operation unit 102 is also provided with an air / water supply button 141, a suction button 142, function buttons 143 to which various functions can be assigned, and an image capture button 144 for receiving image capture instructions (still image, moving image).

[0024] The handheld operation unit 102 is provided with a scope information recording unit 139 for recording individual information (individual information, scope information) of the endoscope 100. The individual information is, for example, the type of the endoscope 100 (direct view or side view, etc.), the model, the individual identification number, the characteristics of the optical system (viewing angle, distortion, etc.), and information on the instruments (treatment tools, etc.) used to treat the subject. The scope information acquisition unit 230 (scope information acquisition unit, individual information acquisition unit; see FIG. 3 ) of the image processing unit 204 acquires this individual information, and uses it for processing by the medical image processing device 200 (image acquisition processing, region of interest recognition processing, instrument information recognition processing, display control processing). The scope information recording unit 139 may be provided in another part, such as inside the light guide connector 108.

[0025] The insertion section 104 is composed of, in order from the hand operation section 102 side, a flexible section 112, a bending section 114, and a tip rigid section 116. That is, the bending section 114 is connected to the base end side of the tip rigid section 116, and the flexible section 112 is connected to the base end side of the bending section 114. The hand operation section 102 is connected to the base end side of the insertion section 104. By operating the hand operation section 102, the user can bend the bending section 114 to change the orientation of the tip rigid section 116 up, down, left, and right. The tip rigid section 116 is provided with an imaging optical system 130, an illumination section 123, a forceps port 126, etc. (see FIGS. 1 and 2).

[0026] During observation and treatment, white light and / or narrowband light (one or more of red narrowband light, green narrowband light, blue narrowband light, and purple narrowband light) can be irradiated from illumination lenses 123A, 123B of illumination unit 123 by operating operation unit 208 (see FIG. 2). In addition, cleaning water is discharged from a water supply nozzle (not shown) by operating air / water supply button 141, and the photographing lens 132 (photographing lens, photographing unit) of photographing optical system 130 and illumination lenses 123A, 123B can be washed. A conduit (not shown) is connected to forceps port 126 opening at tip rigid portion 116, and a treatment tool (not shown) for tumor removal or the like is inserted into this conduit, and the treatment required for the subject can be performed by moving it forward and backward as appropriate.

[0027] As shown in FIGS. 1 and 2, a photographing lens 132 (photographing unit) is disposed on the tip end surface 116A of the tip rigid portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134 (image sensor, image acquisition unit), a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are disposed behind the photographing lens 132, and an image signal is output by these elements. The image sensor 134 is a color image sensor, and has a plurality of pixels constituted by a plurality of light receiving elements arranged in a matrix (two-dimensional array) in a specific pattern array (Bayer array, X-Trans (registered trademark) array, honeycomb array, etc.). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). An image sensor including the image sensor 134, the driving circuit 136, and the AFE 138 in one package may be used. The photographing optical system 130 can generate a color image from pixel signals of three colors, red, green, and blue, or can generate an image from pixel signals of any one or two colors among red, green, and blue. The image sensor 134 may be of an XY address type or a CCD (Charge Coupled Device) type. Each pixel of the image sensor 134 may further include a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0028] An optical image of the subject is formed on the light receiving surface (imaging surface) of the imaging element 134 by the photographing lens 132, converted into an electric signal, and output to the medical image processing device 200 via a signal cable (not shown) and converted into a video signal. As a result, an endoscopic image (observation image, medical image) of the subject is displayed on the screen of a monitor 400 connected to the medical image processing device 200.

[0029] Furthermore, on the tip end surface 116A of the tip hard portion 116, illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographing lens 132. An exit end of a light guide 170 (described later) is disposed behind the illumination lenses 123A and 123B. This light guide 170 is inserted through the insertion portion 104, the handheld operation portion 102, and the universal cable 106, and an entrance end of the light guide 170 is disposed in the light guide connector 108.

[0030] The user can sequentially capture time-series images of the inside of the subject's living body at a determined frame rate (this can be done under the control of the medical image acquisition unit 220) while inserting or removing the endoscope scope 100 (insertion unit 104) configured as described above into or from the subject's living body.

[0031] <Light source configuration> 2, the light source device 300 is composed of an illumination light source 310, an aperture 330, a condenser lens 340, a light source control unit 350, etc., and causes observation light to enter the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a purple light source 310V that respectively irradiate narrowband light of red, green, blue, and purple, and can irradiate narrowband light of red, green, blue, and purple. The illuminance of the observation light from the light source 310 is controlled by the light source control unit 350, and the illuminance of the observation light can be changed (increased or decreased) and the illumination can be stopped as necessary.

[0032] The light source 310 can emit narrowband light of red, green, blue, and purple in any combination. For example, narrowband light of red, green, blue, and purple can be emitted simultaneously to irradiate white light (normal light) as the observation light, or narrowband light (special light) can be irradiated by emitting one or two of them. The light source 310 may further include an infrared light source that irradiates infrared light (an example of narrowband light). In addition, white light or narrowband light may be irradiated as the observation light by using a light source that irradiates white light and a filter that transmits the white light and each narrowband light.

[0033] <Light source wavelength band> The light source 310 may be a light source that generates light in a white band or a plurality of wavelength bands as white band light, or may be a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be a blue band or a green band in the visible range, or a red band in the visible range. When the specific wavelength band is a blue band or a green band in the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and may have a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. When the specific wavelength band is a red band in the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light in the specific wavelength band may have a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0034] The specific wavelength range mentioned above The area is , including a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and Light in a specific wavelength band is The peak wavelength may be in a wavelength range where the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. In this case, the specific wavelength range is 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm to 750 nm. below and Light in a specific wavelength band isThe peak wavelength may be in the wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm or less.

[0035] In addition, the light generated by the light source 310 Wavelength band includes a wavelength band of 790 nm or more and 820 nm or less, or 905 nm or more and 970 nm or less, and The light generated by the light source 310 is The peak wavelength may be in a wavelength band of 790 nm or more and 820 nm or less, or 905 nm or more and 970 nm or less.

[0036] The light source 310 may also include a light source that irradiates excitation light with a peak of 390 nm or more and 470 nm or less. In this case, a medical image (medical image, in vivo image) having information on the fluorescence emitted by a fluorescent substance in a subject (living body) can be obtained. When obtaining a fluorescent image, a dye agent for a fluoroscopy (fluorescein, acridine orange, etc.) may be used.

[0037] It is preferable to configure the type of light source (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. of the light source 310 according to the type and location of the subject, the purpose of observation, etc., and it is also preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc., during observation. When switching the wavelength, the wavelength of the irradiated light may be switched, for example, by rotating a disk-shaped filter (rotary color filter) that is placed in front of the light source and has a filter that transmits or blocks light of a specific wavelength.

[0038] In addition, the imaging element used when carrying out the present invention is not limited to a color imaging element in which a color filter is arranged for each pixel, such as the imaging element 134, but may be a monochrome imaging element. When a monochrome imaging element is used, the wavelength of the observation light can be switched sequentially to capture images in a field sequential (color sequential) manner. For example, the wavelength of the emitted observation light may be switched sequentially between (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (red, green, blue, purple, etc.). Also, one or more narrowband lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched by a rotary color filter (green, blue, purple, etc.). The narrowband light may be infrared light of two or more different wavelengths (first narrowband light, second narrowband light).

[0039] By connecting the light guide connector 108 (see Figures 1 and 2) to the light source device 300, the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A, 123B via the light guide 170, and is irradiated onto the observation range from the illumination lenses 123A, 123B.

[0040] <Configuration of medical image processing device> The configuration of the medical image processing device 200 will be described with reference to Fig. 2. In the medical image processing device 200, an image signal output from the endoscope 100 is input by an image input controller 202, and the image processing unit 204 (processor, computer) performs necessary image processing and outputs the image from a video output unit 206. As a result, an observation image (medical image, endoscopic image, in vivo image) is displayed on a monitor 400 (display device). These processes are performed under the control of a CPU 210 (CPU: Central Processing Unit, processor, computer). A communication control unit 205 controls communication regarding acquisition of medical images between a hospital system (HIS: Hospital Information System) and a hospital LAN (Local Area Network) (not shown), and / or an external system or network.

[0041] <Image processing unit functions> 3 is a functional block diagram of the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220 (medical image acquisition unit), a region of interest recognition unit 222 (region of interest recognition unit), an instrument information recognition unit 224 (instrument information recognition unit), a display control unit 226 (display control unit), a recording control unit 228 (recording control unit), and a scope information acquisition unit 230 (scope information acquisition unit). The processing using these functions will be described in detail later.

[0042] The image processing unit 204 can perform the above-mentioned functions to recognize medical images, determine the biopsy state, calculate features, emphasize or reduce components of a specific frequency band, and emphasize or make inconspicuous a specific target (region of interest, blood vessels at a desired depth, etc.). The image processing unit 204 may include a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating light of a white band or light of a plurality of wavelength bands as white band light. In this case, a signal of a specific wavelength band can be obtained by calculation based on color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) included in the normal light image. The image processing unit 204 may also include a feature image generation unit that generates a feature image by calculation based on at least one of a normal light image obtained by irradiating light of a white band or light of a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light of a specific wavelength band, and acquire and display the feature image as a medical image (medical image). The above-mentioned processing is performed under the control of the CPU 210.

[0043] <Realization of functions using various processors> The functions of each part of the image processing unit 204 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors described above also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacture. When learning and recognizing images as in the present invention, a configuration using a GPU is effective. Furthermore, the various processors described above also include dedicated electric circuits, which are processors having a circuit configuration designed specifically for executing specific processing, such as an ASIC (Application Specific Integrated Circuit).

[0044] The functions of each part may be realized by one processor, or may be realized by multiple processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by one processor. As an example of configuring multiple functions by one processor, first, as represented by a computer, there is a form in which one processor is configured by a combination of one or more CPUs and software, and this processor realizes multiple functions. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system by one IC (Integrated Circuit) chip is used. In this way, various functions are configured by using one or more of the above-mentioned various processors as a hardware structure. Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) that combines circuit elements such as semiconductor elements. These electric circuits may be electric circuits that realize the above-mentioned functions using logical sum, logical product, logical negation, exclusive logical sum, and logical operations combining these.

[0045] When the above-mentioned processor or electric circuit executes software (program), a code readable by the computer (for example, various processors and electric circuits constituting the image processing unit 204, and / or a combination thereof) of the software to be executed is stored in a non-transient recording medium such as a ROM 211 (Read Only Memory) or a flash memory (not shown), and the computer refers to the software. The software stored in the non-transient recording medium includes a program for executing the medical image processing method (method of operating a medical image processing device) according to the present invention and data used during execution (data related to acquisition of medical images, data used to define the biopsy state and set the mode of identification display, parameters used in the recognition unit, etc.). The code may be recorded in a non-transient recording medium such as various types of magneto-optical recording devices and semiconductor memories, instead of the ROM 211. During processing using the software, for example, a RAM 212 (RAM: Random Access Memory) is used as a temporary storage area, and for example, an EEPROM (EEPROM) (not shown) is used as a temporary storage area. Electrically It is also possible to refer to data stored in an erasable and programmable read only memory (ROM). The recording unit 207 may be used as a "non-transitory recording medium".

[0046] The ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable codes of programs that cause the CPU 210 and / or the image processor 204 (computer) to execute various image processing methods (including the medical image processing method according to the present invention). The RAM 212 (Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer during image acquisition. The audio processor 209 outputs messages (audio) related to medical image processing, body part recognition, notification, etc. from the speaker 209A (notification unit, speaker) under the control of the CPU 210 and the image processor 204. The program may be recorded and distributed in an external recording medium (not shown) and installed by the CPU 210 from the recording medium. Alternatively, the program may be stored in a server connected to a network in an externally accessible state, downloaded to the ROM 211 by the CPU 210 upon request, and installed and executed.

[0047] <Operation section> The operation unit 208 can be composed of devices such as a keyboard and a mouse (not shown), and the user can issue instructions to execute a medical image processing method and set conditions necessary for execution (for example, definitions of the treatment status, etc., described below, and the form of identification display) via the operation unit 208.

[0048] <Recognition section using neural network> In the first embodiment, the region of interest recognition unit 222 and the instrument information recognition unit 224 can be configured using a trained model (a model trained using an image set consisting of images of a living body) such as a neural network. The region of interest recognition unit 222 recognizes the region of interest from the observation image (region of interest recognition processing), and the instrument information recognition unit 224 recognizes instrument information (treatment state, pre-treatment state, non-treatment state) from the observation image (instrument information recognition processing). Specifically, the instrument information recognition unit 224 determines whether an instrument is inserted, the amount of insertion, the distance between the instrument and the region of interest, etc. from the observation image, and determines whether the state of the endoscope 100 (medical image processing device 200, endoscope system 10) is a treatment state, a pre-treatment state, or a non-treatment state based on the result. The "treatment state", "pre-treatment state", and "non-treatment state" can be understood, for example, as "a state in which treatment is actually being performed on the region of interest with an instrument (a state in which the user is performing treatment)", "a state in which the user is preparing for treatment, such as by inserting an instrument, or a state in which the instrument is close to the region of interest", and "a state in which no operation for treatment (insertion of an instrument, etc.) is being performed (a state in which the user is not performing treatment or not preparing for it), or a state in which the instrument is far from the region of interest (a state other than the treatment state and the pre-treatment state)". treatment It may be divided into states.

[0049] In addition, "procedure" includes not only biopsy (cutting out an area suspected to be a lesion and sending it for pathology or other testing) but also endoscopic resection such as ESD (Endoscopic Submucosal Dissection) and EMR (Endoscopic Mucosal Resection), and "instruments" includes not only instruments for biopsy but also instruments used for ESD, EMR, etc.

[0050] <Example of recognition unit configuration> The configuration of the recognition unit when performing recognition (detection, discrimination, etc.) using a CNN (Convolutional Neural Network) as a neural network will be described below. FIG. 4 is a diagram showing the configuration of a CNN 562 (neural network). In the example shown in part (a) of FIG. 4, the CNN 562 has an input layer 562A, an intermediate layer 562B, and an output layer 562C. The input layer 562A inputs an endoscopic image (medical image, observation image) acquired by the medical image acquisition unit 220 and outputs a feature amount. The intermediate layer 562B includes a convolution layer 564 and a pooling layer 565, and inputs the feature amount output by the input layer 562A to calculate other feature amounts. These layers have a structure in which a plurality of "nodes" are connected by "edges", and weighting coefficients applied to the input image are associated with the nodes and edges and stored in a weighting coefficient storage unit (not shown). The value of the weighting coefficient changes as learning progresses.

[0051] <Processing in the middle layer> The intermediate layer 562B calculates features by convolution and pooling. The convolution performed in the convolution layer 564 is a process of acquiring a feature map by convolution using a filter, and plays a role in extracting features such as edge extraction from an image. The convolution using this filter generates a "feature map" of one channel (one sheet) for one filter. When downscaling is performed by convolution, the size of the "feature map" becomes smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 565 is a process of reducing (or enlarging) the feature map output by the convolution to create a new feature map, and plays a role in providing robustness so that the extracted features are not affected by translation or the like. The intermediate layer 562B can be configured with one or more layers that perform these processes. Note that the CNN 562 may be configured without the pooling layer 565.

[0052] The CNN 562 may include a fully connected layer 566 as shown in the example of part (b) of Fig. 4. The layer configuration of the CNN 562 is not limited to the case where the convolution layer 564 and the pooling layer 565 are repeated one by one, but may include any layer (for example, the convolution layer 564) in a series of multiple layers.

[0053] 5 is a diagram showing the convolution process by the filter shown in FIG. 4. In the first (first) convolution layer of the intermediate layer 562B, an image set consisting of a plurality of medical images (a learning image set during learning, and a region recognition image set during region recognition) and a filter F 1 The image set is composed of N images (N channels) with an image size of H vertically and W horizontally. When normal light images are input, the images that make up the image set are images of three channels: R (red), G (green), and B (blue). The filter F that is convolved with this image set is 1 Since the image set has N channels (N images), for example, in the case of a filter of size 5 (5 × 5), the filter size will be 5 × 5 × N. This filter F 1 A single filter F 1 A one-channel (one-image) “feature map” is generated for each filter F used in the second convolutional layer. 2 For example, for a filter of size 3 (3×3), the filter size will be 3×3×M.

[0054] Similar to the first convolutional layer, the second to nth convolutional layers use a filter F 2 ~F n The size of the "feature map" in the nth convolutional layer is smaller than that of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0055] Among the layers in intermediate layer 562B, low-order feature extraction (such as edge extraction) is performed in the convolutional layers closer to the input side, and higher-order feature extraction (extraction of features related to the shape, structure, etc. of the object to be recognized) is performed as the layers approach the output side.

[0056] The intermediate layer 562B may include a layer that performs batch normalization in addition to the convolution layer 564 and the pooling layer 565. The batch normalization process is a process that normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in speeding up learning, reducing dependency on initial values, suppressing overlearning, etc.

[0057] The output layer 562C outputs the feature amount calculated by the intermediate layer 562B in a format suitable for part recognition. The output layer 562C may include a fully connected layer.

[0058] The region of interest recognition process by the region of interest recognition unit 222 and the tool information recognition process by the tool information recognition unit 224 may be performed by a common neural network, or may be performed by separate neural networks.

[0059] <Identification of biopsy target area> When observing a wide range of diseases using an endoscope system or the like, a region where the disease is progressing is often biopsied and subjected to a pathological examination. In this case, a medical image processing device detects a biopsy target region (region of interest, etc.) and displays a region suitable for biopsy. However, when a biopsy is performed using an instrument (treatment tool), the doctor knows the biopsy position at the time the instrument is inserted, and if the identification display (discrimination ability) for the target region is maintained in this state (see FIG. 9 described later), there is a risk that the doctor's biopsy will be hindered. Therefore, in the present invention, as will be described in detail below, instrument information in the observation image is recognized, and the observation image is displayed on a display device in a manner in which the biopsy target region (region of interest) has discrimination ability according to the recognition result of the instrument information.

[0060] In the present invention, "discriminability" means "ability of a user, such as a doctor, who visually views an observation image to distinguish a region of interest from other regions." The higher the discrimination ability, the clearer the user can distinguish the region of interest from other regions. Furthermore, "discriminative display" means displaying an observation image with enhanced discrimination ability for the region of interest, and specifically includes highlighting the region of interest itself (e.g., filling in or displaying an outline) and information indicating the region of interest (e.g., displaying a frame surrounding the region of interest, or displaying a figure or symbol indicating the region of interest), as shown in a display example described later.

[0061] <Each step of the medical image processing method> 6 is a flowchart showing the procedure of the medical image processing method (method of operating the medical image processing apparatus) according to the first embodiment. Note that it is assumed that learning of the CNN 562 using the learning data has already been executed.

[0062] <Definition of biopsy status, etc.> The image processing unit 204 sets a definition of the biopsy state and the like in response to a user's operation via the operation unit 208 (step S100: definition setting process). The user can perform the setting operation via a screen 700 (displayed on the monitor 400) illustrated in FIG.

[0063] The screen 700 has areas 702, 710, and 720 in which radio buttons and numerical input areas are arranged. The user can set whether or not to make a judgment based on each item by operating the radio button, and can also input a numerical value that is a judgment criterion for each item. For example, the user can set that "when the amount of instrument insertion is 10 mm or more, it is judged to be a biopsy state (treatment state)" by turning on the radio button 703A of "instrument insertion XX mm or more" in the area 702 (area that defines the biopsy state) and inputting a numerical value (10 mm in the example of FIG. 7) in the area 703B. The instrument information recognition unit 224 can judge that "the instrument and the area of ​​interest overlap" when the distance between the instrument and the area of ​​interest in the observation image is zero or is equal to or less than a threshold value.

[0064] Similarly, the user can set the biopsy state to be determined based on the distance between the instrument and the region of interest, and the use state (operation state) of the instrument. Note that "the instrument is in use (operation state)" includes, for example, the case where the blades of the forceps are open, or the loop of the snare wire is contracted (see Figures 9 to 15 described later). Also, different definitions may be set depending on the type of instrument (forceps, snare, brush, etc.).

[0065] The user can perform such operations for the biopsy state (area 702), the biopsy preparation state (area 710), and the non-biopsy state (area 720). Note that, although Fig. 7 shows an example in which the instrument information is divided into three stages, namely, "biopsy state (treatment state)", "biopsy preparation state (pre-treatment state)", and "non-biopsy state (non-treatment state)", it may be divided into two stages, namely, the biopsy state and the non-biopsy state.

[0066] The user can turn such judgment criteria on / off or input numerical values ​​for multiple items, and when multiple items are turned on, the instrument information recognition unit 224 (processor) can determine that the instrument is in a "biopsy state (or a biopsy preparation state, or a non-biopsy state)" when all of the conditions corresponding to those items are met. In the example of Fig. 7, the instrument information recognition unit 224 can determine that the instrument is in a "biopsy state" when the instrument insertion amount is 10 mm or more, the distance between the instrument and the region of interest is less than 10 mm, and the instrument is in use.

[0067] <Identification display mode settings> The display control unit 226 (processor) sets the mode of the identification display in response to a user operation via the operation unit 208 (step S110: display control step). The user can perform the setting operation via a screen 750 (displayed on the monitor 400) illustrated in FIG.

[0068] As shown in Fig. 8, screen 750 has areas 760, 770, and 780 in which radio buttons are arranged. These areas are areas for setting the manner of identification display in the biopsy state, the biopsy preparation state, and the non-biopsy state, respectively, and the user can set the manner of identification display by operating the radio button in each area. Radio buttons 760A to 760D are provided in area 760 for the biopsy state, and when radio button 760A is turned on, for example, the region of interest is displayed with a dotted outline in the biopsy state. Similarly, the user can set the manner of identification display in the biopsy preparation state and the non-biopsy state with the radio buttons in areas 770 and 780.

[0069] The user can set the mode of identification display such that the discrimination ability of the region of interest is lower in the biopsy state and biopsy preparation state than in the non-biopsy state by operating the radio buttons and the like described above. The user may also set the discrimination ability to be lower in the biopsy state than in the biopsy preparation state. The instrument information recognition unit 224 (processor) may output a warning message when the discrimination ability in the biopsy state and biopsy preparation state is not lower than the discrimination ability in the non-biopsy state in the user's setting, or may set the display mode in the other state in conjunction with the setting of the display mode in one state (discrimination ability of the region of interest is lower in the biopsy state and biopsy preparation state than in the non-biopsy state).

[0070] In this way, in the endoscope system 10 (endoscope system), the user can set the definition of the biopsy state and the mode of the identification display as necessary. The definition of the biopsy state and the mode of the identification display may be set not only at the start of medical image processing, but also at any timing during the processing. Furthermore, the definition of the biopsy state and the mode of the identification display may be set automatically by the endoscope system 10 without the user's operation.

[0071] <Acquiring endoscopic images> The medical image acquisition unit 220 (processor) acquires time-series endoscopic images (observation images, medical images) (step S120: image acquisition step, image acquisition process). The medical image acquisition unit 220 may acquire endoscopic images captured by the endoscope 100, or may acquire endoscopic images recorded in the recording unit 207. The recording control unit 228 can record the acquired endoscopic images in the recording unit 207.

[0072] <Recognition of area of ​​interest and instrument information> The region of interest recognition unit 222 (processor) recognizes the region of interest from the observation image by the CNN 562 (step S130: region of interest recognition step, region of interest recognition processing). In addition, the instrument information recognition unit 224 (processor) recognizes instrument information from the observation image by the CNN 562 (step S140: instrument information recognition step, instrument information recognition processing). The instrument information includes at least one of the following: whether an instrument inserted into a duct communicating with the forceps port 126 of the endoscope 100 is inserted into the subject, the type of instrument inserted into the subject, the length of insertion (the length of the instrument inserted into the subject), the operation state of the instrument inserted into the subject, the distance between the instrument inserted into the subject and the region of interest, and whether the instrument and the region of interest overlap in the observation image. The region of interest recognition unit 222 and the instrument information recognition unit 224 may refer to individual information of the endoscope 100 in the above-mentioned recognition.

[0073] <Determination of biopsy status, etc.> The instrument information recognition unit 224 judges whether it is in a biopsy state or a biopsy preparation state based on the definition and the judgment criterion values ​​set in Fig. 7, and the instrument information recognized in step S140 (step S150: state judgment step). If it is in a biopsy state or a biopsy preparation state (YES in step S150), the display control unit 226 sets the identification display mode of the biopsy state in the case of the biopsy state, and the identification display mode of the biopsy preparation state in the case of the biopsy preparation state, according to the mode set in Fig. 8 (step S160: display control step). If it is in a non-biopsy state (NO in step S150), the display control unit 226 sets the identification display mode of the non-biopsy state in accordance with the mode set in Fig. 8 (step S170: display control step). Note that if the region of interest is not recognized in the first place, there is no need to perform identification display.

[0074] <Identification of observed images> The display control unit 226 (processor) causes the display device to distinctively display the observation image in a manner in which the region of interest has a distinctiveness according to the recognition result of the tool information (step S180: display control step).

[0075] FIG. 9 is an example of identification display of a biopsy target range (region of interest). In the observation image 800 shown in the figure, a region 830 is a true region of interest (the boundary of the region 830 is displayed for convenience of explanation; the same applies below), and a region 850 is a region of interest recognized (detected) by the region of interest recognition unit 222 (CNN 562). In the state shown in the figure, an instrument is not inserted, and it is in a non-biopsy state (non-use state, non-operation state). In the figure, the display control unit 226 displays the region 850 by filling it in, and the identification ability of the region of interest is high, but as described above, if such identification display continues in the biopsy state or biopsy preparation state, it may be an obstacle to the doctor. Therefore, in this embodiment, in the biopsy state and biopsy preparation state, an identification display is performed with a lower identification ability of the region of interest than in the non-biopsy state, as exemplified below. Note that these identification displays can also be performed in the case of procedures other than biopsy (ESD, EMR, etc.).

[0076] (Display example: No. 1) FIG. 10 is a diagram showing an example of an observation image in which the discrimination power of the region of interest is reduced (in a state of preparation for biopsy in which forceps 900, which is an example of an instrument (treatment tool), is inserted). Part (a) of FIG. 10 shows an example of an arrow 852 (symbol) superimposed on a part of the region of interest in the observation image 802, part (b) of the same figure shows an example of an area 850 (region of interest) in an outline display in the observation image 804, and part (c) of the same figure shows an example of a circular figure 854 (figure) superimposed on a part of the region of interest in the observation image 806. The figure to be superimposed is of medium size (see "Partial display (medium)" of the area 770 in FIG. 8). Note that the figures to be superimposed are not limited to the figures and symbols shown in these examples, and may be letters or numbers. The display control unit 226 may also superimpose a frame (such as a bounding box) surrounding the region of interest on the observation image. In the biopsy state, the display control unit 226 can reduce the discrimination ability of the superimposed display compared to the biopsy preparation state by reducing the number of letters, numbers, and symbols to be superimposed, changing solid lines to dotted lines, etc. Furthermore, in addition to reducing the discrimination ability of the superimposed display in the biopsy state, the display control unit 226 may also reduce the discrimination ability of the region of interest compared to the biopsy preparation state by changing the color and / or brightness of the region of interest, etc.

[0077] (Display example: No. 2) Figure 11 shows the equipment and interest region 11A is a diagram showing an example of identification display according to the distance between the forceps 900 (instrument, treatment tool) and the region of interest (true region of interest 832, region of interest 834 recognized by the region of interest recognition unit 222) in the observation image 808. In the state shown in FIG. 11A, the distance between the forceps 900 (instrument, treatment tool) and the region of interest (true region of interest 832, region of interest 834 recognized by the region of interest recognition unit 222) is far, so the region of interest 834 is displayed as an outline. In the state shown in FIG. In the observed image 810 The forceps 900 are close to the area of ​​interest. ,figure The shape 835 (figure) is used to reduce the proportion of the region of interest, thereby reducing the discrimination ability. Note that in part (b) of Fig. 11, the outline of the region of interest 834 is shown by a dotted line, but this dotted line is a reference line to indicate that the proportion of the region of interest is displayed small, and does not need to be actually displayed on the monitor 400.

[0078] (Display example: No. 3) FIG. 12 is a diagram showing an example of identification display according to the operation state of the instrument. In the state shown in part (a) of FIG. 12, the forceps 900A is inserted in the observation image 812 but the blades are closed (non-operation state), so the instrument information recognition unit 224 can determine that the instrument is in a "biopsy preparation state." Therefore, the display control unit 226 displays the region of interest 836 (the region of interest recognized by the region of interest recognition unit 222; the true region of interest is the region of interest 832) in a filled-in state to improve the identification ability. In contrast, in the state shown in part (b) of FIG. 12, the blades of the forceps 900B are open in the observation image 814 (operation state), so the instrument information recognition unit 224 can determine that the instrument is in a "biopsy state," and as a result, the display control unit 226 displays the region of interest 838 in outline form to reduce the identification ability.

[0079] (Display example: No. 4) FIG. 13 is a diagram showing another example of identification display according to the operation state of the instrument. In the state shown in part (a) of FIG. 13, a snare 902 (instrument, treatment tool) is inserted in the observation image 816, but the wire loop 902A is not caught in the region of interest 840 (lesion), so the instrument information recognition unit 224 can determine that the instrument is in a "biopsy preparation state". Therefore, the display control unit 226 superimposes a figure 866A (figure) on the region of interest 840 to increase the identification ability. In contrast, in the state shown in part (b) of FIG. 13, the loop 902A begins to be caught in the region of interest in the observation image 818, so the instrument information recognition unit 224 can determine that the instrument is in a "biopsy state". Therefore, the display control unit 226 superimposes a small circular figure 866B (figure) on the region of interest 840 to decrease the identification ability compared to the state shown in part (a) of the same figure.

[0080] In addition, in the state shown in part (b) of FIG. 13, the tool information recognition unit 224 recognizes that “the tool is interest It may be determined that the area overlaps with the target area (the distance is equal to or smaller than a threshold value).

[0081] (Display example: No. 5) FIG. 14 is a diagram showing yet another example of identification display according to the operation state of the instrument. In the state shown in part (a) of FIG. 14, a snare 902 (instrument, treatment tool) is inserted in the observation image 820 and a wire loop 902A is caught in the region of interest 840 (lesion), but since the loop 902A is in an open state, the instrument information recognition unit 224 can determine that the region is in a "biopsy preparation state". Therefore, the display control unit 226 superimposes a figure 868A (figure) on the region of interest 840 to enhance the identification ability. In contrast, in the state shown in part (b) of FIG. 14, since the loop 902A is about to close in the observation image 822, the instrument information recognition unit 224 can determine that the region is in a "biopsy state". Therefore, the display control unit 226 superimposes an arrow 869A (symbol) and a point 869B (figure, symbol) on the region of interest 840 to reduce the identification ability compared to the state shown in part (a) of the same figure.

[0082] (Display example: No. 6) Fig. 15 is a diagram showing yet another example of identification display according to the operating state of the tool. In the state shown in part (a) of Fig. 15, the brush 904 (tool) is far from the region of interest 842, so the tool information recognition unit 224 can determine that it is in a "biopsy preparation state", and the display control unit 226 superimposes a figure 870A (figure) on the region of interest 842 to enhance identification. In contrast, in the state shown in part (b) of Fig. 15, the brush 904 is close to the region of interest 842 in the observation image 826, so the tool information recognition unit 224 can determine that it is in a "biopsy state", and the display control unit 226 superimposes a figure 870B (figure) on the region of interest 842 to enhance identification. 842 8A. An arrow 870B (symbol) and a point 870C (figure, symbol) are superimposed on the image 870, making the discrimination ability lower than that shown in part (a) of the same figure.

[0083] In addition, in the above-mentioned display example, the display control unit 226 may reduce the ability to distinguish between the biopsy state and the biopsy preparation state by changing the color and / or brightness of the fill, outline, symbol, etc. compared to the non-biopsy state.

[0084] The CPU 210 and image processing unit 204 repeat the processes of steps S120 to S180 until the observation is completed (while NO is returned in step S190).

[0085] As described above, according to the medical image processing device, endoscopic system, medical image processing method, and medical image processing program of the first embodiment, the user can set the definition and identification display mode of the biopsy state, etc. as necessary, the instrument information recognition unit 224 recognizes the instrument information, and the display control unit 226 performs identification display of the observation image based on the recognition result, thereby allowing the area of ​​interest to be displayed with appropriate identification ability.

[0086] (Additional Note) In addition to the above-mentioned aspects, the following configurations are also included within the scope of the present invention.

[0087] (Appendix 1) The medical image analysis processing unit detects an area of ​​interest that is an area that should be noted based on pixel features of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0088] (Appendix 2) The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the pixel feature amount of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0089] (Appendix 3) The medical image analysis result acquisition unit: Obtained from a recording device that records the results of medical image analysis; The analysis results are a region of interest, which is an area of ​​interest contained in a medical image, and / or the presence or absence of an object of interest.

[0090] (Appendix 4) A medical image processing apparatus in which the medical image is a normal light image obtained by irradiating a white band light or light of a plurality of wavelength bands as white band light.

[0091] (Appendix 5) A medical image is an image obtained by irradiating light of a specific wavelength band. A medical imaging device, wherein the specific wavelength band is a narrower band than the white wavelength band.

[0092] (Appendix 6) A medical image processing device, wherein the specific wavelength band is the blue or green band of the visible range.

[0093] (Appendix 7) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

[0094] (Appendix 8) The specific wavelength band is the red band of the visible range for medical imaging processing devices.

[0095] (Appendix 9) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0096] (Appendix 10) A medical image processing device, wherein the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients.

[0097] (Appendix 11) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the specific wavelength band has a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

[0098] (Appendix 12) Medical images are in vivo images that show the inside of a living body. In vivo images are obtained by a medical image processing device that has information on the fluorescence emitted by fluorescent substances within a living body.

[0099] (Appendix 13) The fluorescence peaks at 390 nm A medical image processing device that obtains images by irradiating the inside of a living body with excitation light having a wavelength of 470 nm or more and 470 nm or less.

[0100] (Appendix 14) Medical images are in vivo images that show the inside of a living body. The particular wavelength band is an infrared wavelength band for medical imaging devices.

[0101] (Appendix 15) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less, or a wavelength band of 905 nm or more and 970 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less, or a wavelength band of 905 nm or more and 970 nm or less.

[0102] (Appendix 16) the medical image acquisition unit includes a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating white light or light of a plurality of wavelength bands as white light; Medical images are special light images taken by a medical image processing device.

[0103] (Appendix 17) A medical image processing device in which signals of a specific wavelength band are obtained by calculation based on the RGB or CMY color information contained in a normal light image.

[0104] (Appendix 18) a feature image generating unit that generates a feature image by calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as the white band light, and a special light image obtained by irradiating light in a specific wavelength band; A medical image processing device processes medical images as feature images.

[0105] (Appendix 19) A medical image processing device according to any one of claims 1 to 18, an endoscope that acquires an image by irradiating at least one of light in a white wavelength band and light in a specific wavelength band; An endoscope apparatus comprising:

[0106] (Appendix 20) A diagnostic support device comprising the medical image processing device according to any one of appendixes 1 to 18.

[0107] (Appendix 21) A medical service support device comprising the medical image processing device according to any one of appendixes 1 to 18.

[0108] Although the embodiment and other examples of the present invention have been described above, the present invention is not limited to the above-mentioned aspects, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0109] 10 Endoscope System 100 Endoscope 102 Handheld operation unit 104 Insertion part 106 Universal Cable 108 Light Guide Connector 112 Soft part 114 Curved section 116 Hard tip 116A Tip side end surface 123 Lighting Department 123A Lighting Lens 123B Lighting Lens 126 Forceps mouth 130 Photographing Optical System 132 Photographic Lens 134 Image sensor 136 Drive circuit 138 AFE 139 Scope Information Recording Unit 141 Air / water supply button 142 Suction button 143 Function button 144 Shooting button 170 Light Guide 200 Medical image processing device 202 Image Input Controller 204 Image Processing Unit 205 Communication control section 206 Video output section 207 Recording Department 208 Operation section 209 Audio Processing Unit 209A Speaker 210 CPU 211 ROM 212 RAM 220 Medical Image Acquisition Department 222 Region of interest recognition unit 224 Instrument information recognition section 226 Display control section 228 Recording control section 230 Scope information acquisition unit 300 Light source device 310 light source 310B Blue light source 310G green light source 310R red light source 310V purple light source 330 Aperture 340 Condenser Lens 350 Light source control unit 400 Monitors 562A Input Layer 562B Middle Tier 562C output layer 564 convolutional layers 565 Pooling Layer 566 fully connected layer 700 screens 702 area 703A Radio Button 703B area 710 area 720 areas 750 screens 760 areas 760A Radio Button 760B Radio Button 760C Radio Button 760D Radio Button 770 area 780 areas 800 Observation images 802 Observation images 804 Observation images 806 Observation images 808 Observation images 812 Observation images 814 Observation images 816 Observation images 818 Observation images 822 Observation images 826 Observation images 830 areas 832 Areas of Interest 834 Areas of Interest 835 Shapes 836 Areas of Interest 838 Areas of Interest 840 Areas of Interest 842 Areas of Interest 850 areas 852 Arrow 854 Shapes 866A Shape 866B Shape 868A Shape 869A Arrow 869B points 870A Shape 870B Arrow 870C points 900 forceps 900A Forceps 900B forceps 902 Snare 902A wheel 904 Brush F 1 filter F 2 filter S100~S190 Each step of the medical image processing method

Claims

1. 1. A medical imaging device comprising a processor, The processor, An image acquisition process for acquiring an observation image of the subject; A region of interest recognition process for recognizing a region of interest from the observation image; an instrument information recognition process for recognizing, from the observation image, instrument information that is information on an instrument used to treat the subject; a display control process for displaying the observation image on a display device in a manner in which the region of interest has a discrimination ability corresponding to the recognition result of the tool information; Do the following: In the instrument information recognition process, a state of treatment by the instrument is determined based on the instrument information; In the display control process, the distinguishable display is performed based on a result of the determination. Medical imaging equipment.

2. The processor, In the tool information recognition process, a state of the treatment by the tool is determined based on the tool information as being a treatment state in which the treatment is being performed on the region of interest, a pre-treatment state in which the treatment is being prepared, or a non-treatment state which is a state other than the treatment state and the pre-treatment state; The medical image processing apparatus according to claim 1 , wherein, in the display control process, the observation images are displayed with a lower discrimination power in the treatment state and the pre-treatment state than in the non-treatment state.

3. The processor, The medical image processing apparatus according to claim 2 , wherein, in the display control process, the observation image is displayed with the discrimination power lowered in the treatment state than in the pre-treatment state.

4. The processor, 4. The medical image processing device according to claim 2 or 3, wherein in the instrument information recognition processing, the state is determined to be one of the treatment state, the pre-treatment state, and the non-treatment state based on the instrument information including at least one of whether the instrument is inserted, the type of the inserted instrument, the length of the insertion, the operation state of the instrument, the distance between the instrument and the region of interest, and whether the instrument and the region of interest overlap in the observation image.

5. The processor, The medical image processing apparatus according to claim 2 , wherein in the display control process, a frame surrounding the region of interest is superimposed on the observation image in the treatment state and / or the pre-treatment state.

6. The processor, The medical image processing apparatus according to claim 2 , wherein in the display control process, a symbol indicating the region of interest is superimposed on the observation image in the treatment state and the pre-treatment state.

7. The processor, The medical image processing device according to claim 2 , wherein in the display control process, at least one of a character, a figure, and a symbol is superimposed on a part of the region of interest in the observation image in the treatment state and the pre-treatment state.

8. The processor, The medical image processing apparatus according to claim 5 , wherein the discrimination ability of the superimposed display is reduced in the treatment state and the pre-treatment state compared to the non-treatment state.

9. The processor, 9. The medical image processing device according to claim 2, wherein, in the display control processing, the observation image is displayed by changing a color and / or brightness of the region of interest in the treatment state and the pre-treatment state compared to the non-treatment state.

10. A medical image processing apparatus according to any one of claims 1 to 9, the display device for displaying the observed image; an endoscope to be inserted into the subject, the endoscope having an imaging unit that captures the observation image; An endoscope system comprising:

11. A method of operating a medical imaging device having a processor, comprising: The processor, an image acquiring step of acquiring an observation image of a subject; a region of interest recognition step of recognizing a region of interest from the observation image; an instrument information recognition step of recognizing, from the observation image, instrument information that is information on an instrument used to treat the subject; a display control step of displaying the observation image on a display device in such a manner that the region of interest has a discrimination ability corresponding to the recognition result of the tool information; Run In the instrument information recognition step, a state of treatment by the instrument is determined based on the instrument information; The display control step performs the identification display based on the result of the judgment.

12. The processor, In the tool information recognition step, it is determined whether the region of interest is in a treatment state in which the tool is performing a treatment on the region of interest, a pre-treatment state in which the region of interest is being prepared for the treatment, or a non-treatment state which is a state other than the treatment state and the pre-treatment state, based on the tool information; The operating method according to claim 11 , wherein in the display control step, the observation image is displayed on the display device with the discrimination ability lowered in the treatment state and the pre-treatment state than in the non-treatment state.

13. The processor, The operating method according to claim 12, wherein in the display control step, the observation image is displayed with the discrimination power lowered in the treatment state than in the pre-treatment state.

14. The processor, The operation method according to claim 12 or 13, in which the instrument information recognition process determines whether the state is the treatment state, the pre-treatment state, or the non-treatment state based on the instrument information including at least one of the following: whether the instrument is inserted, the type of the inserted instrument, the length of the insertion, the operation state of the instrument, the distance between the instrument and the area of ​​interest, and whether the instrument and the area of ​​interest overlap in the observation image.

15. A medical image processing program causing a computer to carry out the method according to any one of claims 11 to 14.

Citation Information

Patent Citations

  • Endoscope apparatus

    JP2016064281A

  • Thoracic imaging, distance measuring, surgical awareness, and notification system and method

    US20200188033A1

  • Endoscope system

    WO2020036224A1