Medical image processing apparatus, medical image processing method, endoscope system, and recording medium

Abstract

This invention provides a medical image processing apparatus, endoscope system, medical image processing method, and recording medium capable of displaying a region of interest with appropriate recognition power. One aspect of the medical image processing apparatus of this invention is a medical image processing apparatus equipped with a processor, which performs: image acquisition processing to acquire an observation image of a subject; region of interest identification processing to identify a region of interest from the observation image; instrument information identification processing to identify information about the instruments used in the treatment of the subject, i.e., instrument information, from the observation image; and display control processing to identify and display the observation image on a display device in a manner that the region of interest has a recognition power corresponding to the recognition result of the instrument information.

Description

TECHNICAL FIELD

[0001] The present application relates to a medical image processing apparatus, a medical image processing method, an endoscope system, and a recording medium. BACKGROUND

[0002] The endoscope system described in Patent Literature 1 is provided with an image acquisition section that acquires an image obtained by photographing an object, an identification section that performs identification processing for identifying the object using the image, a determination section that determines an operation with respect to the object, a setting section that sets the identification section to be valid or invalid using a determination result of the determination section, and a notification section that notifies of a valid or invalid state of the identification section.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: WO 2020 / 036224 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] An embodiment of the present application provides a medical image processing apparatus, a medical image processing method, an endoscope system, and a medical image processing program that can display a region of interest with appropriate identification ability.

[0008] MEANS FOR SOLVING THE PROBLEMS

[0009] A medical image processing apparatus according to a first aspect includes a processor that performs: image acquisition processing that acquires an observation image of a subject; region of interest identification processing that identifies a region of interest from the observation image; instrument information identification processing that identifies instrument information of an instrument used for treatment of the subject from the observation image; and display control processing that displays the observation image on a display device with the region of interest having an identification ability corresponding to a result of identification of the instrument information.

[0010] A medical image processing apparatus according to a second aspect includes the processor of the first aspect, wherein, in the instrument information identification processing, the processor determines, based on the instrument information, which of a treatment state in which the region of interest is treated by the instrument, a pre-treatment state in which preparation for treatment is performed, the treatment state and the pre-treatment state, and a non-treatment state other than the treatment state and the pre-treatment state, and in the display control processing, the processor displays the observation image with the identification ability being lower in the treatment state and the pre-treatment state than in the non-treatment state.

[0011] A medical image processing apparatus according to a third aspect includes the processor of the second aspect, wherein, in the display control processing in the treatment state, the processor displays the observation image with the identification ability being lower in the treatment state than in the pre-treatment state.

[0012] In the second or third method, the medical image processing device involved in the fourth method determines, in the instrument information recognition processing, whether it is a treatment state, a pre-treatment state, or a non-treatment state based on instrument information including at least one of the following: whether an instrument is inserted, the type of inserted instrument, the length of insertion, the operating state of the instrument, the distance between the instrument and the area of ​​interest, and whether the instrument and the area of ​​interest overlap in the observed image.

[0013] In any of the second to fourth methods, the medical image processing apparatus involved in the fifth method, during display control processing, in the treatment state and / or pre-treatment state, overlays a box around the region of interest on the observed image.

[0014] In any of the second to fifth methods, the medical image processing apparatus involved in the sixth method, during the display control processing, in the treatment state and the pre-treatment state, displays a symbol representing the area of ​​interest overlaid on the observed image.

[0015] In any of the second to sixth methods, during the display control process, in the treatment state and the pre-treatment state, the processor displays at least one of characters, graphics, and symbols on the observed image, overlapping a portion of the area of ​​interest.

[0016] In any of the fifth to seventh methods, the medical image processing device involved in the eighth method, in the treatment state and the pre-treatment state, compared with the non-treatment state, reduces the recognition power of the overlay display.

[0017] In any of the second to eighth methods, the medical image processing apparatus involved in the ninth method, during the display control processing, in the treatment state and the pre-treatment state, compared with the non-treatment state, the processor changes the color and / or brightness of the area of ​​interest to display the observed image.

[0018] The endoscopic system according to the tenth method includes: a medical image processing device according to any one of the first to ninth methods; a display device for displaying observation images; and an endoscopic observer inserted into the subject and having a photographic section for taking observation images.

[0019] The medical image processing method according to the eleventh aspect of the present invention enables a computer to perform: an image acquisition step, acquiring an observation image of a subject; a region of interest identification step, identifying a region of interest from the observation image; an instrument information identification step, identifying information about the instruments used in the treatment of the subject, i.e., instrument information, from the observation image; and a display control step, identifying and displaying the observation image on a display device in such a way that the region of interest has a recognition capability corresponding to the recognition result of the instrument information. The medical image processing methods according to the eleventh aspect and the following aspects can also be understood as operating methods of a medical image processing apparatus.

[0020] The medical image processing method involved in the twelfth method, in the eleventh method, in the instrument information recognition process, determines, based on the instrument information, which of the following states is being treated by the instrument on the area of ​​interest: treatment state, pre-treatment state, or state other than treatment state and pre-treatment state, i.e., non-treatment state. In the display control process, in the treatment state and pre-treatment state, the recognition power is reduced compared to the non-treatment state, and the observed image is displayed on the display device.

[0021] The medical image processing method involved in the thirteenth method, in the twelfth method, in the display control process, in the treatment state, reduces the recognition power compared to the pre-treatment state and displays the observed image.

[0022] The medical image processing method involved in the fourteenth method, in the twelfth or thirteenth method, in the device information recognition process, determines which of the following is the treatment state, the pre-treatment state, and the non-treatment state based on device information including at least one of the following: whether a device is inserted, the type of inserted device, the length of insertion, the operating state of the device, the distance between the device and the area of ​​interest, and whether the device and the area of ​​interest overlap in the observed image.

[0023] The medical image processing program according to the fifteenth method causes a computer to execute the medical image processing method according to any one of the eleventh to fourteenth methods. A non-transitory recording medium containing computer-readable code of the medical image processing program according to the fifteenth method can also be cited as an aspect of the present invention. Attached Figure Description

[0024] Figure 1 This is an external view of the endoscope system according to the first embodiment.

[0025] Figure 2 This is a block diagram representing the essential structure of an endoscope system.

[0026] Figure 3 This is a functional block diagram of the image processing unit.

[0027] Figure 4This is a diagram representing the structure of a convolutional neural network.

[0028] Figure 5 This is a diagram illustrating the case of filter-based convolution processing.

[0029] Figure 6 This is a flowchart illustrating the steps of the medical image processing method according to the first embodiment.

[0030] Figure 7 This is an example image showing a screen that defines the settings, such as the handling state.

[0031] Figure 8 This is an example image showing a screen displaying a set identification method.

[0032] Figure 9 This is a diagram showing an example of the identification display of the target area (region of interest) for a biopsy.

[0033] Figure 10 This is a diagram illustrating an example of reducing the recognition power of the region of interest to display the observed image.

[0034] Figure 11 This is a diagram showing an example of an identification display corresponding to the distance between the appliance and the area of ​​interest.

[0035] Figure 12 This is a diagram showing an example of an identification display corresponding to the operating status of an appliance.

[0036] Figure 13 This is another example of a diagram showing the identification display corresponding to the operating status of an appliance.

[0037] Figure 14 This is another example of an identification display corresponding to the operating status of an appliance.

[0038] Figure 15 This is another example of an identification display corresponding to the operating status of an appliance. Detailed Implementation

[0039] Hereinafter, with reference to the accompanying drawings, the embodiments of the medical image processing apparatus, medical image processing method, endoscope system and medical image processing program involved in the present invention will be described in detail.

[0040] <First Implementation Method>

[0041] <Structure of an Endoscopic System>

[0042] Figure 1 This is an external view of the endoscope system 10 (endoscope system). Figure 2 This is a block diagram showing the essential structure of the endoscope system 10. For example... Figure 1 ,2 As shown, the endoscope system 10 consists of an endoscope observer 100 (image acquisition unit, endoscope observer), a medical image processing device 200 (medical image processing device, processor, medical image acquisition unit, area of ​​interest recognition unit, instrument information recognition unit, display control unit, recording control unit), a light source device 300 (light source device), and a monitor 400 (display device, display).

[0043] <Structure of the Endoscopic Observation Device>

[0044] The endoscope observation device 100 includes a handheld operating unit 102 and an insertion unit 104 connected to the handheld operating unit 102. The operator (user) holds the handheld operating unit 102 and operates it, inserting the insertion unit 104 into the body of the subject (organism) for observation. In addition, the handheld operating unit 102 is provided with an air / water supply button 141, a suction button 142, function buttons 143 assigned with various functions, and a photography button 144 for receiving photographic instructions (still image, moving image).

[0045] The handheld operation unit 102 is equipped with an observer information recording unit 139 that records individual information (individual information, observer information) of the endoscope observer 100. Individual information includes, for example, the type of endoscope observer 100 (direct or side-view, etc.), model, individual identification number, characteristics of the optical system (angle of view, deformation, etc.), and information on the instruments used for the treatment of the patient (treatment instruments, etc.). The image processing unit 204 includes an observer information acquisition unit 230 (observer information acquisition unit, individual information acquisition unit; see reference). Figure 3 The individual's information is acquired for processing by the medical image processing device 200 (image acquisition processing, region of interest recognition processing, instrument information recognition processing, display control processing). Furthermore, the observer information recording unit 139 may also be located within the optical connector 108 or other parts.

[0046] The insertion part 104 is composed of a flexible part 112, a curved part 114, and a rigid tip part 116, sequentially from the side of the hand-operated part 102. Specifically, the curved part 114 is connected to the base end of the rigid tip part 116, and the flexible part 112 is connected to the base end of the curved part 114. The hand-operated part 102 is connected to the base end of the insertion part 104. The user can bend the curved part 114 and change the direction of the rigid tip part 116 by operating the hand-operated part 102. The rigid tip part 116 is equipped with a photographic optical system 130, an illumination part 123, a clamping mouth 126, etc. (see reference). Figure 1 , 2 ).

[0047] During observation and handling, it can be done through the operation unit 208 (refer to...). Figure 2The illumination unit 123 emits white light and / or narrowband light (one or more of red, green, blue, and purple narrowband light) from the illumination lenses 123A and 123B. Additionally, the operation of the air / water supply button 141 releases cleaning water from a water supply nozzle (not shown) to clean the imaging lens 132 (imaging lens, imaging unit) and illumination lenses 123A and 123B of the imaging optical system 130. A conduit (not shown) connects to a clamping port 126 opening at the top rigid section 116. A treatment instrument (not shown) for tumor resection, etc., is inserted into this conduit, allowing it to be moved appropriately to perform the necessary treatment on the subject.

[0048] like Figure 1 , 2 As shown, a photographic lens 132 (photographic unit) is disposed on the top side end face 116A of the top rigid part 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134 (image sensor, image acquisition unit), a driving circuit 136, and an AFE 138 (AFE: Analog Front End) are disposed inside the photographic lens 132, and image signals are output through these elements. The image sensor 134 is a color image sensor, comprising multiple pixels composed of multiple light-receiving elements arranged in a matrix in a specific pattern (Bayer arrangement, X-Trans arrangement, honeycomb arrangement, etc.) (two-dimensional arrangement). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). An image sensor that includes the image sensor 134, the driving circuit 136, and the AFE 138 in a single package can also be used. The photographic optical system 130 can generate a color image based on pixel signals of red, green, and blue, or it can generate an image based on pixel signals of any one or two of the three colors. Furthermore, the image sensor 134 can be an XY address type or a CCD (Charge Coupled Device) type. Additionally, each pixel of the image sensor 134 can also have a violet color filter corresponding to the violet light source 310V and / or an infrared filter corresponding to an infrared light source.

[0049] The optical image of the subject is imaged onto the light-receiving surface (image-receiving surface) of the imaging element 134 through the photographic lens 132 and converted into an electrical signal. This signal is then output to the medical image processing device 200 via a signal cable (not shown) and converted into an image signal. As a result, the endoscopic image (observation image, medical image) of the subject is displayed on the monitor 400 connected to the medical image processing device 200.

[0050] Additionally, on the top side end face 116A of the top rigid portion 116, illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographic lens 132. On the inner side of the illumination lenses 123A and 123B, the emission end of the light guide 170 (described later) is provided. The light guide 170 is inserted into the insertion portion 104, the hand operation portion 102, and the universal cable 106. The incident end of the light guide 170 is disposed in the light guide connector 108.

[0051] By inserting or removing the endoscope 100 (insertion part 104) of the above structure into the biological body of the subject, the user can take pictures at a determined frame rate (which can be controlled by the medical image acquisition unit 220) and sequentially capture time-series images of the biological body.

[0052] <Structure of the Light Source Device>

[0053] like Figure 2 As shown, the light source device 300 comprises a light source 310 for illumination, an aperture 330, a condenser lens 340, and a light source control unit 350, etc., allowing observation light to be incident on the light guide 170. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively illuminate narrow bands of red, green, blue, and violet light, and can illuminate narrow bands of red, green, blue, and violet light. The illuminance of the observation light based on the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illuminance of the observation light and stop illumination as needed.

[0054] The light source 310 can emit narrowband light of red, green, blue, and violet in any combination. For example, it can emit narrowband light of red, green, blue, and violet simultaneously to illuminate white light (ordinary light) as observation light, or it can emit any one or two of these narrowband lights to illuminate narrowband light (special light). The light source 310 may also be equipped with an infrared light source to illuminate infrared light (an example of narrowband light). Alternatively, it can illuminate white light or narrowband light as observation light by using a light source that illuminates white light and filters that transmit white light and each narrowband light.

[0055] <wavelength band of the light source>

[0056] Light source 310 can be light in the white frequency band, or a light source that generates light in multiple wavelength bands as white frequency band light, or a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band can be the blue or green band of the visible range, or the red band of the visible range. When the specific wavelength band is the blue or green band of the visible range, it can also contain wavelength bands between 390 nm and 450 nm or between 530 nm and 550 nm, and have a peak wavelength within these wavelength bands. Alternatively, when the specific wavelength band is the red band of the visible range, it can also contain wavelength bands between 585 nm and 615 nm or between 610 nm and 730 nm, and the light in the specific wavelength band has a peak wavelength within these wavelength bands.

[0057] The aforementioned specific wavelength band may also include wavelength bands with different absorption coefficients in oxidized hemoglobin and deoxidized hemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength bands with different absorption coefficients in oxidized hemoglobin and deoxidized hemoglobin. In this case, the specific wavelength band may also include wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light in the specific wavelength band has a peak wavelength in the wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0058] In addition, the wavelength band of the light generated by the light source 310 may also include a wavelength band of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength band of 790nm to 820nm or 905nm to 970nm.

[0059] Alternatively, the light source 310 may also be equipped with an excitation light source with a peak illumination of 390 nm to 470 nm. In this case, medical images (medical images, in vivo images) containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be acquired. When acquiring fluorescence images, fluorescent dyes (fluorescein, acridine orange, etc.) can also be used.

[0060] The type of light source 310 (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), wavelength, and presence or absence of filters are preferably configured according to the type and location of the subject, the purpose of observation, etc. Furthermore, during observation, it is preferable to combine and / or switch the wavelength of the observation light according to the type and location of the subject, the purpose of observation, etc. When switching wavelengths, for example, the wavelength of the irradiated light can also be switched by rotating a disc-shaped filter (rotating color filter) that has a filter positioned in front of the light source that transmits or blocks light of a specific wavelength.

[0061] Furthermore, the imaging element used in implementing this invention is not limited to a color imaging element like imaging element 134, which equips each pixel with a color filter; a monochrome imaging element can also be used. When using a monochrome imaging element, the wavelengths of the observation light can be switched sequentially to capture images in a surface order (color order). For example, the wavelengths of the emitted observation light can be switched sequentially between (violet, blue, green, red), or the wavelengths of the observation light emitted by rotating the color filter (red, green, blue, violet, etc.) after irradiating broadband light (white light) can be switched. Alternatively, the wavelengths of the observation light emitted by rotating the color filter (green, blue, violet, etc.) after irradiating one or more narrowband lights (green, blue, violet, etc.) can be switched. The narrowband light can also be infrared light with two or more different wavelengths (first narrowband light, second narrowband light).

[0062] By using the optical guide connector 108 (reference) Figure 1 , 2 The light source device 300 is connected to the observation light source device 300. The observation light emanating from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and then irradiates the observation area from the illumination lenses 123A and 123B.

[0063] <Structure of Medical Image Processing Device>

[0064] based on Figure 2The structure of the medical image processing apparatus 200 will be described. The medical image processing apparatus 200 receives the image signal output from the endoscope observer 100 via the image input controller 202, performs necessary image processing by the image processing unit 204 (processor, computer), and outputs it from the video output unit 206. Thus, the observed image (medical image, endoscopic image, image inside a biological body) is displayed on the monitor 400 (display device). This processing is performed under the control of the CPU 210 (CPU: Central Processing Unit, processor, computer). The communication control unit 205 performs communication control for acquiring medical images, etc., with a hospital information system (HIS) or a hospital LAN (Local Area Network) not shown, and / or external systems or networks.

[0065] <Functions of the Image Processing Unit>

[0066] Figure 3 This is a functional block diagram of the image processing unit 204. The image processing unit 204 includes a medical image acquisition unit 220, a region of interest recognition unit 222, an instrument information recognition unit 224, a display control unit 226, a recording control unit 228, and an observer information acquisition unit 230. The processing using their functions will be described in detail below.

[0067] The image processing unit 204 can perform the above-described functions to recognize medical images, determine biopsy status, calculate feature quantities, emphasize or reduce components in specific frequency bands, and emphasize or ignore specific objects (areas of interest, blood vessels at desired depths, etc.). The image processing unit 204 may also include a special light image acquisition unit, which acquires a special light image with information of a specific wavelength band based on a normal light image obtained by illuminating light in the white frequency band, or by illuminating light in multiple wavelength bands as white frequency band light. In this case, the signal of the specific wavelength band can be obtained by calculation based on the color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) contained in the normal light image. Additionally, the image processing unit 204 may also include a feature image generation unit, which acquires and displays a feature image as a medical image (medical image). This feature image generation unit generates the feature image through calculations based on at least one of a general light image obtained by irradiating light in the white frequency band, or light in multiple wavelength bands as the white frequency band, and a special light image obtained by irradiating light in a specific wavelength band. Furthermore, the above processing is performed under the control of the CPU 210.

[0068] <Implementation of the functions of various processors>

[0069] The functions of each part of the image processing unit 204 described above can be implemented using various processors and recording media. Among these processors are, for example, general-purpose processors that execute software (programs) to achieve various functions, namely CPUs (Central Processing Units). Additionally, among the aforementioned processors are image processing-specific processors such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and other processors whose circuit structures can be modified after manufacturing, i.e., programmable logic devices (PLDs). When performing image learning or recognition as described in this invention, using a GPU structure is effective. Furthermore, processors with circuit structures specifically designed for performing specific processing, such as ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits, are also included among the aforementioned processors.

[0070] The functions of each part can be implemented by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, a single processor can implement multiple functions. As examples of implementing multiple functions with a single processor, firstly, there are processors, such as computers, which use a combination of one or more CPUs and software to form a single processor and implement it for multiple functions. Secondly, there are processors, such as System-on-Chip (SoC), which use a single IC (Integrated Circuit) chip to implement the overall system functions. In this way, one or more of the above-mentioned processors are used as hardware structures to implement various functions. More specifically, the hardware structure of these various processors is a circuit composed of circuit elements such as semiconductor components. These circuits can also be circuits that use logical operations such as AND, AND, AND, NOT, XOR, and combinations thereof to implement the aforementioned functions.

[0071] When the processor or circuit described above executes the software (program), the code of the software to be executed, which can be read by a computer (e.g., various processors or circuits constituting the image processing unit 204, and / or combinations thereof), is stored in advance in a non-transitory recording medium such as ROM 211 (ROM: Read Only Memory) or flash memory (not shown), and the computer refers to this software. The software stored in advance in the non-transitory recording medium includes a program for executing the medical image processing method (operation method of the medical image processing device) according to the present invention and data used during execution (data related to the acquisition of medical images, data used for defining or identifying display methods such as biopsy status, parameters used by the recognition unit, etc.). The code may also be recorded in a non-transitory recording medium such as various optical magnetic recording devices or semiconductor memories instead of ROM 211. When processing using the software, RAM 212 (RAM: Random Access Memory) may be used as a temporary storage area, for example, and data stored in EEPROM (Electrically Erasable and Programmable Read Only Memory, not shown) may also be referenced. The recording unit 207 can also be used as a "non-temporary recording medium".

[0072] Additionally, ROM 211 (Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code that enables the CPU 210 and / or image processing unit 204 (computer) to execute various image processing methods (including the medical image processing method according to this invention). RAM 212 (Random Access Memory) is a storage element used for temporary storage during various processing operations, and can also be used as a cache during image acquisition. The voice processing unit 209, under the control of the CPU 210 and image processing unit 204, outputs messages (voice) related to medical image processing, site recognition, and notification from the speaker 209A (notification unit, speaker). Furthermore, the program can be recorded and allocated to an external recording medium (not shown), and installed from that recording medium by the CPU 210. Alternatively, the program can be stored in an externally accessible state on a server connected to a network, and downloaded to ROM 211 by the CPU 210 upon request, and then installed and executed.

[0073] Operations Department

[0074] The operation unit 208 may be composed of devices such as a keyboard and mouse (not shown). Users can use the operation unit 208 to give instructions for the execution of medical image processing methods or to set the conditions required for execution (e.g., the definition of treatment status or identification display method described later).

[0075] <Recognition Department Using Neural Networks>

[0076] In the first embodiment, a learning model (a model learned using an image set composed of images obtained from photographs of organisms) is used to construct the region of interest identification unit 222 and the instrument information identification unit 224. The region of interest identification unit 222 identifies the region of interest from the observed image (region of interest identification processing), and the instrument information identification unit 224 identifies instrument information (treatment state, pre-treatment state, non-treatment state) from the observed image (instrument information identification processing). Specifically, the instrument information identification unit 224 determines whether an instrument is inserted or the amount of insertion, the distance between the instrument and the region of interest, etc., based on the observed image, and determines, based on the results, whether the state of the endoscope observer 100 (medical image processing device 200, endoscope system 10) is the treatment state, the pre-treatment state, or the non-treatment state. Furthermore, "treatment state," "pre-treatment state," and "non-treatment state" can be understood as, for example, "the state in which the area of ​​interest is actually treated using an instrument (the state in which the user is treating)," "the state in which the user is preparing for treatment, such as inserting an instrument, or the state in which the instrument is close to the area of ​​interest," "the state in which no operation for treatment (such as inserting an instrument) is performed (the state in which the user has not performed treatment or preparation)," or "the state in which the instrument is far from the area of ​​interest (a state other than the treatment state and the pre-treatment state)." Alternatively, the treatment state and the pre-treatment state can be distinguished as treatment state and non-treatment state without separating them.

[0077] In addition, "treatment" includes not only biopsy (removal of lesions and suspicious areas for pathological examination, etc.), but also endoscopic resections such as ESD (Endoscopic Submucosal Dissection) or EMR (Endoscopic Mucosal Resection). "Instruments" include not only instruments used for biopsy, but also instruments used for ESD or EMR, etc.

[0078] <Structure example of the identification section>

[0079] The following section explains the structure of the recognition unit when using a CNN (Convolutional Neural Network) as a neural network for recognition (detection, identification, etc.). Figure 4 This is a diagram representing the structure of CNN562 (a neural network). Figure 4In the example shown in part (a), the CNN562 has an input layer 562A, an intermediate layer 562B, and an output layer 562C. The input layer 562A takes into account the endoscopic image (medical image, observation image) acquired by the medical image acquisition unit 220 and outputs feature values. The intermediate layer 562B includes a convolutional layer 564 and a pooling layer 565, takes the feature values ​​output by the input layer 562A as input, and calculates other feature values. These layers form a structure where multiple "nodes" are connected by "edges," and the weighting coefficients applied to the input image are stored in a weighting coefficient storage unit (not shown) in association with the nodes and edges. The values ​​of the weighting coefficients change as learning progresses.

[0080] <Processing in the intermediate layer>

[0081] Intermediate layer 562B calculates feature values ​​through convolution and pooling operations. The convolution operation performed by convolutional layer 564 is a process of obtaining feature maps using convolution operations on filters, responsible for feature extraction such as edge detection. By using the convolution operation on the filter, a "feature map" for one channel (one image) is generated for each filter. The size of the "feature map" decreases as convolution is performed across layers during downscaling. The pooling operation performed by pooling layer 565 is a process of shrinking (or expanding) the feature maps output by the convolution operations to form new feature maps, responsible for providing robustness to the extracted features, preventing them from being affected by parallel shifts, etc. Intermediate layer 562B can consist of one or more layers performing these processes. Furthermore, CNN562 can also be constructed without pooling layer 565.

[0082] CNN562 can also be like Figure 4 The example shown in part (b) includes a fully connected layer 566. The layer structure of CNN562 is not limited to the case of convolutional layer 564 and pooling layer 565 being repeated layer by layer, but can also include multiple layers of any kind (e.g., convolutional layer 564) consecutively.

[0083] Figure 5 It means Figure 4The diagram illustrates the convolution process performed on the filter. In the first convolutional layer of the intermediate layer 562B, a convolution operation is performed between an image set consisting of multiple medical images (a learning image set during learning and a site recognition image set during site recognition) and filter F1. The image set consists of N images (N channels) with image dimensions H and W. When a normal light image is input, the images constituting the image set are images from three channels: R (red), G (green), and B (blue). Regarding filter F1, which performs the convolution operation with this image set, since the image set has N channels (N images), for example, if it is a filter of size 5 (5×5), the filter size becomes 5×5×N. By using the convolution operation of filter F1, a "feature map" for one channel (one image) is generated for each filter F1. When the filter F2 used in the second convolutional layer is, for example, a filter of size 3 (3×3), the filter size becomes 3×3×M.

[0084] Similar to the first convolutional layer, filters F2 to F1 are used in the second to nth convolutional layers. n The convolution operation. The size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it has been downscaled by the convolutional or pooling layers up to the previous layer.

[0085] In the middle layer 562B, low-order feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers near the input side, and high-order feature extraction (extraction of features related to the shape, structure, etc. of the object to be identified) is performed as the layers move closer to the output side.

[0086] In addition to the convolutional layer 564 and the pooling layer 565, the intermediate layer 562B may also include a layer for batch normalization. Batch normalization is a process that normalizes the distribution of data in small batches during learning, which serves to accelerate learning, reduce dependence on initial values, and suppress overlearning.

[0087] The output layer 562C outputs the feature quantities calculated by the intermediate layer 562B in a form consistent with part recognition. The output layer 562C may also include a fully connected layer.

[0088] Furthermore, the attention area identification processing performed by the attention area identification unit 222 and the appliance information identification processing performed by the appliance information identification unit 224 can be performed through a common neural network or through individual neural networks.

[0089] <Identification and display of biopsy target area>

[0090] When using endoscopic systems to observe extensive diseases, biopsies are often taken from areas of disease progression for pathological examination. In this case, medical image processing devices detect the target biopsy area (region of interest, etc.) and display the area suitable for biopsy. However, when using instruments (treatment instruments) for biopsies, the physician knows the biopsy location at the moment of insertion. Maintaining the recognition and display of the target area even in this state (recognition capability) is crucial (see below). Figure 9 This could hinder the doctor's biopsy. Therefore, in this invention, as will be explained in detail below, instrument information in the observed image is identified, and the observed image is displayed on a display device in such a way that the biopsy target area (area of ​​interest) has a recognition power corresponding to the identification result of the instrument information.

[0091] Furthermore, in this invention, "recognition ability" refers to "the ability of a user, such as a doctor, to visually identify an image and distinguish the area of ​​interest from other areas." The higher the user's recognition ability, the more clearly they can distinguish the area of ​​interest from other areas. Additionally, "recognition display" refers to displaying the observed image with enhanced recognition ability of the area of ​​interest. Specifically, as shown in the display examples described later, this includes emphasizing the area of ​​interest itself (e.g., filling, outlining) or displaying information representing the area of ​​interest (e.g., displaying a box surrounding the area of ​​interest, displaying a graphic or symbol representing the area of ​​interest).

[0092] <Various Processing Steps in Medical Image Processing>

[0093] Figure 6 This is a flowchart illustrating the steps of the medical image processing method (operation method of the medical image processing apparatus) according to the first embodiment. Furthermore, it is presented as a flowchart showing that learning using the CNN562 with training data has already been performed.

[0094] <Definition of biopsy status, etc.>

[0095] The image processing unit 204 sets definitions for biopsy status, etc., based on user operation via the operation unit 208 (step S100: definition setting process). The user can then... Figure 7 The setup operation is performed on the illustrated screen 700 (displayed on monitor 400).

[0096] Screen 700 has areas 702, 710, and 720 configured with radio buttons and numerical input areas. Users can use the radio buttons to set whether to perform judgments based on various items, and can also input values ​​that serve as the judgment criteria for each item. For example, a user can activate the radio button 703A for "Instrument insertion XX mm or more" in area 702 (the area defining the biopsy status), and input a value in area 703B (in... Figure 7In the example, the measurement is 10mm, and the setting is "when the instrument insertion depth is 10mm or more, it is judged as a biopsy state (treatment state)". In addition, when the distance between the instrument and the area of ​​interest in the observed image is zero or below a threshold, the instrument information recognition unit 224 can determine that "the instrument overlaps with the area of ​​interest".

[0097] Similarly, users can set the biopsy status to be determined based on the distance between the instrument and the area of ​​interest, as well as the instrument's usage status (operation status). Furthermore, "instrument in use (operation status)" includes, for example, the opening of the forceps' blades or the retraction of the snare's loops (see below). Figures 9-15 Additionally, different definitions can be set according to the type of tool (pliers, snares, brushes, etc.).

[0098] Users can perform this operation on the biopsy status (area 702), biopsy preparation status (area 710), and non-biopsy status (area 720). Furthermore, in Figure 7 The text shows an example of dividing instrument information into three stages: "biopsy status (treatment status)," "biopsy preparation status (pre-treatment status)," and "non-biopsy status (non-treatment status)," but it can also be divided into two stages: biopsy status and non-biopsy status.

[0099] Users can enable / disable or input values ​​for multiple items to establish this judgment criterion. When multiple items are enabled, and all conditions corresponding to these items are met, the instrument information recognition unit 224 (processor) can determine that it is a biopsy state (or a biopsy preparation state, or a non-biopsy state). Figure 7 In the example, when the instrument insertion depth is 10mm or more, the distance between the instrument and the area of ​​interest is less than 10mm, and the instrument is in use, the instrument information recognition unit 224 can determine that "it is a biopsy state".

[0100] <Display Recognition Method Settings>

[0101] The display control unit 226 (processor) sets the display recognition mode based on user operation via the operation unit 208 (step S110: display control process). The user can then... Figure 8 The setup operation is performed on the illustrated screen 750 (displayed on monitor 400).

[0102] like Figure 8As shown, screen 750 has areas 760, 770, and 780 equipped with radio buttons. These areas are used to set the recognition display mode in biopsy status, biopsy preparation status, and non-biopsy status, respectively. Users can set the recognition display mode by operating the radio buttons in each area. Radio buttons 760A to 760D are provided in area 760 concerning biopsy status. For example, if radio button 760A is turned on, the area of ​​interest is displayed with a dashed outline in biopsy status. Similarly, users can use the radio buttons in areas 770 and 780 to set the recognition display mode in biopsy preparation status and non-biopsy status.

[0103] Users can configure the recognition display mode via the radio buttons mentioned above: In biopsy and biopsy preparation states, the recognition power of the area of ​​interest is reduced compared to the non-biopsy state. Alternatively, users can configure the recognition power to be reduced in biopsy state compared to the biopsy preparation state. The instrument information recognition unit 224 (processor) can, according to the user's settings, output a warning message if the recognition power in biopsy and biopsy preparation states is not lower than that in the non-biopsy state, or it can automatically configure the display mode in other states (reducing the recognition power of the area of ​​interest in biopsy and biopsy preparation states compared to the non-biopsy state) if a display mode is configured in any state.

[0104] Thus, in the endoscope system 10 (endoscope system), the user can set the definition and display method of biopsy status, etc., as needed. Furthermore, the setting of the definition and display method of biopsy status, etc., can be performed not only at the start of medical image processing, but also at any time during processing. Alternatively, the endoscope system 10 can automatically set the definition and display method of biopsy status, etc., without relying on user operation.

[0105] <Acquisition of endoscopic images>

[0106] The medical image acquisition unit 220 (processor) acquires time-series endoscopic images (observation images, medical images) (step S120: image acquisition process, image acquisition handling). The medical image acquisition unit 220 can acquire endoscopic images captured by the endoscope observer 100, or endoscopic images recorded in the recording unit 207. The recording control unit 228 can record the acquired endoscopic images in the recording unit 207.

[0107] <Identification of Areas of Interest and Equipment Information>

[0108] The region of interest identification unit 222 (processor) identifies the region of interest from the observed image using CNN 562 (step S130: region of interest identification process, region of interest identification processing). Additionally, the instrument information identification unit 224 (processor) uses CNN 562 to identify instrument information from the observed image (step S140: instrument information identification process, instrument information identification processing). Instrument information includes at least one of the following: whether an instrument inserted into the tubing connected to the endoscope observation device 100 and communicating with the forceps port 126 is inserted into the patient body; the type of instrument inserted into the patient body; the length of insertion (length of the instrument inserted into the patient body); the operating state of the instrument inserted into the patient body; the distance between the instrument inserted into the patient body and the region of interest; and whether the instrument and the region of interest overlap in the observed image. The region of interest identification unit 222 and the instrument information identification unit 224 may also refer to individual information of the endoscope observation device 100 in the above identification process.

[0109] <Assessment of biopsy status, etc.>

[0110] The appliance information identification unit 224 is based on... Figure 7 The system uses the defined criteria and numerical values ​​as the judgment standard, along with the instrument information identified in step S140, to determine whether the device is in biopsy status or biopsy preparation status (step S150: status judgment process). When it is in biopsy status or biopsy preparation status (YES in step S150), the display control unit 226 proceeds according to the criteria set in the system. Figure 8 The display control unit 226 sets the display mode according to the settings in the text. When the biopsy status is active, the display mode for identifying the biopsy status is set; when the biopsy preparation status is active, the display mode for identifying the biopsy preparation status is set (step S160: display control process). When the status is not active (NO in step S150), the display control unit 226... Figure 8 The method set in the middle determines the identification and display mode for non-biopsy states (step S170: display control process). In addition, when the area of ​​interest is not originally identified, identification and display are not required.

[0111] <Observation of Image Recognition and Display>

[0112] The display control unit 226 (processor) identifies and displays the observed image on the display device in such a way that the area of ​​interest has the recognition power corresponding to the recognition result of the appliance information (step S180: display control process).

[0113] Figure 9 This is an example of the identification and display of the target area (region of interest) for a biopsy. Figure 9 In the observed image 800, region 830 is the true region of interest (the boundary of region 830 is shown for illustrative purposes; the same applies below), and region 850 is the region of interest identified (detected) by the region of interest recognition unit 222 (CNN562).Figure 9 In the indicated state, the instrument is not inserted; it is a non-biopsy state (non-use state, non-operation state). Figure 9 In this embodiment, the display control unit 226 fills the area 850 to display the area of ​​interest, resulting in high recognition accuracy. However, as mentioned above, continuing this recognition display during biopsy or biopsy preparation could become an obstacle for the physician. Therefore, in this embodiment, during biopsy and biopsy preparation, a recognition display is performed that reduces the recognition accuracy of the area of ​​interest compared to the non-biopsy state, as illustrated below. Furthermore, this recognition display can also be performed in the same manner during procedures other than biopsy (such as ESD or EMR).

[0114] (Example shown: First)

[0115] Figure 10 This is an example of reducing the recognizable area of ​​interest to display the observed image (an example of an instrument (treatment instrument), namely the biopsy preparation state of forceps 900). Figure 10 Part (a) is an example of arrow 852 (symbol) displayed in the observed image 802, overlapping with a portion of the region of interest. Figure 10 Part (b) is an example of outlining region 850 (region of interest) in observed image 804. Figure 10 Part (c) illustrates an example of a circular graphic 854 (graphic) that overlaps with a portion of the region of interest in the observed image 806. The overlapping graphic is medium-sized (see reference). Figure 8 The overlapping display is in the "partial display (center)" of region 770. Furthermore, the overlapping display is not limited to the graphics or symbols shown in these examples; it can also be characters or numbers. Additionally, the display control unit 226 can also overlay a box (boundary box, etc.) surrounding the region of interest on the observed image. In the biopsy state, the display control unit 226 can reduce the recognition power of the overlapping display compared to the biopsy preparation state by reducing the number of characters or numbers or symbols in the overlapping display, or by setting solid lines to dashed lines. Furthermore, in addition to reducing the recognition power of the overlapping display in the biopsy state, the display control unit 226 can also reduce the recognition power of the region of interest compared to the biopsy preparation state by changing the color and / or brightness of the region of interest.

[0116] (Example shown: Part Two)

[0117] Figure 11 This is a diagram illustrating an example of an identification display corresponding to the distance between the device and the area of ​​interest. Figure 11 In the state shown in part (a), in the observed image 808, the distance between the pliers 900 (instrument, handling instrument) and the region of interest (the actual region of interest 832, the region of interest 834 identified by the region of interest recognition unit 222) is relatively far, therefore the region of interest 834 is outlined, but...Figure 11 In the state shown in part (b), the distance between the clamps 900 and the area of ​​interest is relatively close in the observed image 810. Therefore, the proportion of the area of ​​interest is reduced by the graphic 835 (graphic), which decreases the recognition ability. Furthermore, in Figure 11 In part (b), the outline of the area of ​​interest 834 is represented by a dashed line, but this dashed line is a reference line used to indicate that the area of ​​interest is displayed at a reduced scale, so it does not actually need to be displayed on the monitor 400.

[0118] (Example shown: Thirdly)

[0119] Figure 12 This is a diagram illustrating an example of an identification display corresponding to the operating status of an appliance. Figure 12 In the state shown in part (a), the forceps 900A is inserted into the observation image 812, but the cutting edge is closed (non-operation state). Therefore, the instrument information recognition unit 224 can determine that it is a "biopsy preparation state". Therefore, the display control unit 226 fills the area of ​​interest 836 (the area of ​​interest recognized by the area of ​​interest recognition unit 222; the actual area of ​​interest is area of ​​interest 832) for display, improving recognition accuracy. In contrast, in Figure 12 In the state shown in part (b), the blade of the pliers 900B is open (operation state) in the observed image 814. Therefore, the instrument information recognition unit 224 can determine that "it is a biopsy state". As a result, the display control unit 226 sets the area of ​​interest 838 to outline display to reduce the recognition power.

[0120] (Example shown: Part Four)

[0121] Figure 13 This is another example of a display showing the identification of the operating status of an appliance. Figure 13 In the state shown in part (a), a snare 902 (instrument, treatment instrument) is inserted into the observation image 816, but the thread reel 902A is not engaged in the area of ​​interest 840 (lesion). Therefore, the instrument information recognition unit 224 can determine that "it is a biopsy preparation state". Therefore, the display control unit 226 overlays the graphic 866A (graphic) in the area of ​​interest 840 to improve recognition accuracy. In contrast, in Figure 13 In the state shown in part (b), in the observed image 818, wheel 902A begins to engage with the area of ​​interest, so the instrument information recognition unit 224 can determine that "it is a biopsy state". Therefore, the display control unit 226 overlays a smaller circular graphic 866B (graphic) in the area of ​​interest 840, making the recognition more accurate than before. Figure 13 The state shown in part (a) is reduced.

[0122] In addition, Figure 13In the state shown in part (b), the appliance information recognition unit 224 can also determine that "the appliance overlaps with the area of ​​interest (the distance is below the threshold)".

[0123] (Example shown: Fifth)

[0124] Figure 14 This is another example of a display showing the identification of the operating status of an appliance. Figure 14 In the state shown in part (a), a snare 902 (instrument, treatment instrument) is inserted into the observation image 820, and the thread reel 902A is engaged in the area of ​​interest 840 (lesion), but the reel 902A is in an open state. Therefore, the instrument information recognition unit 224 can determine that "it is a biopsy preparation state". Therefore, the display control unit 226 overlays the graphic 868A (graphic) in the area of ​​interest 840 to improve recognition accuracy. In contrast, in Figure 14 In the state shown in part (b), when viewing image 822, wheel 902A is about to close, so the instrument information recognition unit 224 can determine that "it is a biopsy state". Therefore, the display control unit 226 overlays arrow 869A (symbol) and dot 869B (graphic, symbol) in the area of ​​interest 840, making the recognition more accurate than before. Figure 14 The state shown in part (a) is reduced.

[0125] (Example shown: Item 6)

[0126] Figure 15 This is another example of a display showing the identification of the operating status of an appliance. Figure 15 In the state shown in part (a), in the observed image 824, the brush 904 (instrument) is far from the area of ​​interest 842. Therefore, the instrument information recognition unit 224 can determine that "it is a biopsy preparation state," and the display control unit 226 overlays the graphic 870A (graphic) in the area of ​​interest 842 to improve recognition accuracy. In contrast, in Figure 15 In the state shown in part (b), when viewing image 826, brush 904 is close to the area of ​​interest 842, so the instrument information recognition unit 224 can determine that "it is a biopsy state", and the display control unit 226 overlays arrow 870B (symbol) and dot 870C (graphic, symbol) in the area of ​​interest 842, making the recognition power higher than that of the previous state. Figure 15 The state shown in part (a) is reduced.

[0127] Furthermore, in the above-mentioned display example, the display control unit 226 can also reduce the recognition ability of the biopsy state and biopsy preparation state by changing the color and / or brightness of the fill, outline, symbols, etc., compared with the non-biopsy state.

[0128] CPU210 and image processing unit 204 repeat steps S120 to S180 until the observation ends (during the period when it is NO in step S190).

[0129] As described above, according to the medical image processing apparatus, endoscope system, medical image processing method and medical image processing program involved in the first embodiment, the user can set the definition and recognition display mode of biopsy status, etc. as needed, the instrument information recognition unit 224 recognizes the information of the instrument, and the display control unit 226 performs recognition display of the observed image based on the recognition result, thereby enabling the display of the area of ​​interest with appropriate recognition power.

[0130] (Postscript)

[0131] In addition to the methods described above, the structures described below are also included within the scope of this invention.

[0132] (Postscript 1)

[0133] A medical image processing apparatus, wherein a medical image analysis and processing unit detects a region of interest, i.e., a region of interest, based on the feature quantities of pixels in a medical image.

[0134] The Medical Image Analysis Result Acquisition Department acquires the analysis results from the Medical Image Analysis and Processing Department.

[0135] (Postscript 2)

[0136] A medical image processing apparatus, wherein a medical image analysis and processing unit detects the presence or absence of an object of interest based on the feature quantities of pixels in a medical image.

[0137] The Medical Image Analysis Result Acquisition Department acquires the analysis results from the Medical Image Analysis and Processing Department.

[0138] (Note 3)

[0139] A medical image processing device, wherein a medical image analysis result acquisition unit

[0140] Acquired from a recording device that records the analysis results of medical images.

[0141] The analysis result is either or both of the regions of interest and the presence or absence of objects of interest contained in the medical image.

[0142] (Note 4)

[0143] A medical image processing apparatus, wherein the medical image is a common light image obtained by irradiating light in the white frequency band, or by irradiating light in multiple wavelength bands as white frequency band light.

[0144] (Note 5)

[0145] A medical image processing apparatus, wherein the medical image is an image obtained by irradiating light of a specific wavelength band.

[0146] A specific wavelength band is a band narrower than the white wavelength band.

[0147] (Note 6)

[0148] A medical image processing device, wherein a specific wavelength band is the blue or green band of the visible range.

[0149] (Note 7)

[0150] A medical image processing device, wherein a specific wavelength band includes a wavelength band of 390nm to 450nm or 530nm to 550nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band of 390nm to 450nm or 530nm to 550nm.

[0151] (Note 8)

[0152] A medical image processing device, wherein a specific wavelength band is the red band of the visible range.

[0153] (Note 9)

[0154] A medical image processing device, wherein a specific wavelength band includes a wavelength band of 585nm to 615nm or 610nm to 730nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band of 585nm to 615nm or 610nm to 730nm.

[0155] (Postscript 10)

[0156] A medical image processing apparatus, wherein a specific wavelength band includes wavelength bands with different absorption coefficients in oxidized hemoglobin and deoxygenated hemoglobin, and the light in the specific wavelength band has a peak wavelength in the different wavelength bands with different absorption coefficients in oxidized hemoglobin and deoxygenated hemoglobin.

[0157] (Postscript 11)

[0158] A medical image processing device, wherein a specific wavelength band includes wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light in the specific wavelength band has a peak wavelength in the wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0159] (Postscript 12)

[0160] A medical image processing device, wherein the medical images are images of a living organism taken within the organism.

[0161] Images within a living organism contain information about the fluorescence emitted by fluorescent substances within that organism.

[0162] (Postscript 13)

[0163] A medical image processing device, wherein fluorescence can irradiate an organism with excitation light having a peak value of 390 nm or higher and 470 nm or lower.

[0164] (Postscript 14)

[0165] A medical image processing device, wherein the medical images are images of a living organism taken within the organism.

[0166] The specific wavelength band is the wavelength band of infrared light.

[0167] (Postscript 15)

[0168] A medical image processing device, wherein a specific wavelength band includes a wavelength band of 790nm to 820nm or 905nm to 970nm, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 790nm to 820nm or 905nm to 970nm.

[0169] (Postscript 16)

[0170] A medical image processing apparatus includes a medical image acquisition unit comprising a special light image acquisition unit. This special light image acquisition unit acquires a special light image containing information of a specific wavelength band based on a normal light image obtained by illuminating light in the white frequency band, or by illuminating light in multiple wavelength bands as white frequency band light.

[0171] Medical images are special light images.

[0172] (Postscript 17)

[0173] A medical image processing apparatus, wherein a signal in a specific wavelength band is obtained by calculation based on the RGB or CMY color information contained in a normal light image.

[0174] (Postscript 18)

[0175] A medical image processing apparatus includes a feature quantity image generation unit that generates a feature quantity image by performing calculations based on at least one of a general light image obtained by irradiating light in the white frequency band, or light in multiple wavelength frequency bands irradiated by light in the white frequency band, and a special light image obtained by irradiating light in a specific wavelength frequency band.

[0176] Medical images are characteristic images.

[0177] (Postscript 19)

[0178] An endoscope device comprising:

[0179] The medical image processing apparatus described in any one of Appendices 1 to 18; and

[0180] An endoscope acquires images by illuminating light in a white wavelength band or at least one of a specific wavelength band.

[0181] (Postscript 20)

[0182] A diagnostic aid device comprising any one of the appendices 1 to 18.

[0183] (Postscript 21)

[0184] A medical business support device comprising any one of the appendices 1 to 18.

[0185] The embodiments and other examples of the present invention have been described above, but the present invention is not limited to the above-described methods, and various modifications can be made without departing from the spirit of the present invention.

[0186] Symbol Explanation

[0187] 10 Endoscopic Systems

[0188] 100 Endoscopic Observation Devices

[0189] 102 Hands-on Operations Department

[0190] 104 Insertion Section

[0191] 106 General Purpose Cable

[0192] 108 Optical Wire Connector

[0193] 112 Soft parts

[0194] 114 Bend

[0195] 116 Hardened tip

[0196] 116A Top Side Face

[0197] 123 Lighting Department

[0198] 123A Illumination Lens

[0199] 123B Illumination Lens

[0200] 126 Pliers opening

[0201] 130 photographic optical system

[0202] 132 Photographic Lens

[0203] 134 camera elements

[0204] 136 drive circuit

[0205] 138 AFE

[0206] 139 Observer Information Recording Department

[0207] 141 Gas and water supply buttons

[0208] 142 Attraction Button

[0209] 143 Function Buttons

[0210] 144 Camera button

[0211] 170 optical waveguide

[0212] 200 Medical Image Processing Device

[0213] 202 Image Input Controller

[0214] 204 Image Processing Department

[0215] 205 Communications Control Department

[0216] 206 Video Output Department

[0217] 207 Records Department

[0218] 208 Operations Department

[0219] 209 Speech Processing Department

[0220] 209A Speaker

[0221] 210 CPU

[0222] 211 ROM

[0223] 212 RAM

[0224] 220 Medical Image Acquisition Department

[0225] 222 Area of ​​Concern Identification Department

[0226] 224 Equipment Information Identification Department

[0227] 226 Display Control Unit

[0228] 228 Record Control Department

[0229] 230 Observer Information Acquisition Department

[0230] 300 Light Source Device

[0231] 310 Light Source

[0232] 310B Blue Light Source

[0233] 310G Green Light Source

[0234] 310R Red Light Source

[0235] 310V purple light source

[0236] 330 aperture

[0237] 340 Condensing Lens

[0238] 350 Light Source Control Unit

[0239] 400 monitor

[0240] 562A Input Layer

[0241] 562B Intermediate Layer

[0242] 562C Output Layer

[0243] 564 convolutional layers

[0244] 565 Pooling Layer

[0245] 566 Fully Connected Layer

[0246] 700 screens

[0247] Area 702

[0248] 703A Radio Button

[0249] Area 703B

[0250] Area 710

[0251] Area 720

[0252] 750 frames

[0253] 760 area

[0254] 760A Radio Button

[0255] 760B Radio Button

[0256] 760C Radio Button

[0257] 760D radio buttons

[0258] Area 770

[0259] Area 780

[0260] 800 Observation Image

[0261] 802 Observe the image

[0262] 804 Observe the image

[0263] 806 Observe the image

[0264] 808 Observe the image

[0265] 812 Observe the image

[0266] 814 Observe the image

[0267] 816 Observe the image

[0268] 818 Observe the image

[0269] 822 Observe the image

[0270] 826 Observe the image

[0271] Area 830

[0272] 832 Area of ​​Concern

[0273] 834 Area of ​​Concern

[0274] 835 Graphics

[0275] 836 Area of ​​Concern

[0276] 838 Area of ​​Concern

[0277] 840 Area of ​​Concern

[0278] 842 Area of ​​Concern

[0279] Area 850

[0280] 852 arrow

[0281] 854 graphics

[0282] 866A Graphics

[0283] 866B graphics

[0284] 868A Graphics

[0285] 869A Arrow

[0286] Point 869B

[0287] 870A Graphics

[0288] 870B Arrow

[0289] 870C point

[0290] 900 Pliers

[0291] 900A Pliers

[0292] 900B Pliers

[0293] 902 Trap

[0294] 902A Wheel

[0295] 904 brush

[0296] F1 filter

[0297] F2 filter

[0298] Steps of the medical image processing method (S100-S190)

Claims

1. A medical image processing device, comprising a processor, wherein, The processor performs: Image acquisition and processing to acquire observation images of the subject; Region of interest identification processing: Identifying regions of interest from the observed image; Instrument information identification processing identifies information about the instruments used to treat the subject from the observed image, i.e., instrument information. as well as The display control process identifies and displays the observed image on the display device in a manner that the region of interest has a recognition capability corresponding to the recognition result of the instrument information. The processor, In the device information identification and processing, based on the device information, it is determined which of the following states is being processed: a processing state where the area of ​​interest is being processed using the device; a pre-processing state where preparation for the processing is being made; the processing state; or a state other than the pre-processing state, i.e., a non-processing state. In the display control process, the observed image is displayed with the recognition power reduced compared to the non-disposal state, in both the disposal state and the pre-disposal state.

2. The medical image processing apparatus according to claim 1, wherein, In the display control process, in the disposal state, the processor reduces the recognition power compared to the pre-disposal state and displays the observed image.

3. The medical image processing apparatus according to claim 1 or 2, wherein, In the device information recognition processing, the processor determines which of the following is the treatment state, the pre-treatment state, and the non-treatment state based on device information including at least one of the following: whether the device is inserted, the type of the inserted device, the length of the insertion, the operating state of the device, the distance between the device and the area of ​​interest, and whether the device and the area of ​​interest overlap in the observed image.

4. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, in the disposal state and / or the pre-disposal state, the processor overlays a box around the region of interest onto the observed image.

5. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, in the processing state and the pre-processing state, the processor overlays a symbol representing the region of interest onto the observed image.

6. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, in the disposal state and the pre-disposal state, the processor displays at least one of characters, graphics, and symbols overlapping a portion of the area of ​​interest on the observed image.

7. The medical image processing apparatus according to claim 4, wherein, In the processing state and the pre-processing state, compared with the non-processing state, the processor reduces the recognition power of the overlay display.

8. The medical image processing apparatus according to claim 1 or 2, wherein, In the display control process, compared with the non-processing state in the processing state and the pre-processing state, the processor changes the color and / or brightness of the area of ​​interest to display the observed image.

9. An endoscope system comprising: The medical image processing apparatus according to any one of claims 1 to 8; The display device displays the observed image; and An endoscopic observer, which is inserted into the subject, has a camera unit for taking images of the observed objects.

10. A medical image processing method that enables a computer to: The image acquisition process involves acquiring observation images of the subject. The region of interest identification process identifies the region of interest from the observed image; The instrument information identification process identifies information about the instruments used to treat the subject from the observed image, namely, instrument information. as well as The display control process identifies and displays the observed image on a display device in such a way that the area of ​​interest has a recognition capability corresponding to the recognition result of the instrument information. In the device information identification process, based on the device information, it is determined which of the following states is being processed: a processing state where the area of ​​interest is being processed using the device; a pre-processing state where preparation for the processing is being made; or a state other than the processing state and the pre-processing state, i.e., a non-processing state. In the display control process, in the processing state and the pre-processing state, the recognition power is reduced compared to the non-processing state, and the observed image is displayed on the display device.

11. The medical image processing method according to claim 10, wherein, In the display control process, in the processing state, the recognition power is reduced compared to the pre-processing state to display the observed image.

12. The medical image processing method according to claim 10 or 11, wherein, In the device information recognition process, based on device information including at least one of the following: whether the device is inserted, the type of the inserted device, the length of the insertion, the operating state of the device, the distance between the device and the area of ​​interest, and whether the device and the area of ​​interest overlap in the observed image, it is determined whether it is the treatment state, the pre-treatment state, or the non-treatment state.

13. A non-transitory, computer-readable recording medium containing a program that causes a computer to perform the medical image processing method according to any one of claims 10 to 12.

Images