Medical image processing device, endoscope system, medical image processing system, operation method of medical image processing device, program, and storage medium

The medical image processing device enhances the reliability of AI-based classification by visually separating the classification-relevant regions from the original image, allowing observers to validate the appropriateness of classification results.

JP7583100B2Active Publication Date: 2024-11-13FUJIFILM CORP
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2023071374
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-07-09
Filing Date
2023-04-25
Publication Date
2024-11-13
Estimated Expiration
2039-06-14

AI Technical Summary

Technical Problem

Existing AI-based automatic classification systems for medical images, such as endoscopic images, often produce inappropriate results when lesions are hidden, blurry, or otherwise unsuitable for classification, and it is difficult for observers to determine the appropriateness of these results.

Method used

A medical image processing device that classifies images into multiple classes, generates a region image highlighting the areas contributing to classification, and displays this region image separately from the original image, allowing observers to assess the appropriateness of the classification.

Benefits of technology

Enables observers to understand which areas of the image contributed to the classification and whether the classification is appropriate, thereby improving the reliability of automatic classification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583100000003
    Figure 0007583100000003
  • Figure 0007583100000004
    Figure 0007583100000004
  • Figure 0007583100000005
    Figure 0007583100000005
Patent Text Reader

Abstract

To provide a medical image processing apparatus, an endoscope system, a medical image processing system, a method of operating the medical image processing apparatus, a program and a storage medium that allow an observer to grasp whether classification is appropriate in automatic classification of medical images.SOLUTION: A medical image processing apparatus 14 includes: an image acquisition unit (40) that acquires a captured image (38) generated by imaging a living body; a classification unit (48) that classifies the captured image into two or more classes; an image generation unit (50) that generates a region image visualizing a location of a region that contributes to the classification using the classification unit in the captured image classified by the classification unit; and a display signal transmission unit (44) that transmits to a display device a display signal representing the captured image, a classification result derived using the classification unit, and the region image, where the display signal transmission unit transmits to the display device a display signal for displaying the region image separately from the captured image.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a medical image processing device, an endoscope system, a medical image processing system, an operating method for a medical image processing device, a program, and a storage medium, and more particularly to automatic classification of lesions in medical images. [Background technology]

[0002] In the medical field, examinations are performed using modalities such as endoscope systems. In recent years, a technology has been known that analyzes medical images such as endoscopic images, which are time-series images captured using an endoscope scope, automatically classifies lesions contained in the endoscopic images, and provides the classification results. In this specification, image classification and image discrimination are treated as the same concept.

[0003] Patent Document 1 describes an endoscopic image processing system that supports the detection of characteristic lesion sites in images captured using an endoscope. When the endoscopic image processing system described in the document executes a lesion estimation function, it displays the location of the lesion site estimated to be the lesion site in the image displayed on the monitor by surrounding it with a dotted line.

[0004] Patent Document 2 describes an image processing system that performs texture analysis on the overall shading of a captured image in which blood vessels are captured, and uses the results of the texture analysis to perform classification according to pathological diagnosis, such as non-tumor, adenoma, etc. The document also describes an example in which the probability of correct classification is displayed.

[0005] Patent Document 3 describes an endoscope device that removes images unsuitable for diagnosis from images of the inside of a body cavity captured using the endoscope device.

[0006] Patent Document 4 describes an image processing device that processes time-series images of a living body lumen captured in time series using a medical observation device. The image processing device described in this document determines a normal mucous membrane region in the images that constitute the time-series images as a specific region. The image processing device described in this document also calculates the reliability of the specific region. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 5528255 [Patent Document 2] JP 2017-70609 A [Patent Document 3] Patent No. 4615963 [Patent Document 4] Patent No. 5576711 Summary of the Invention [Problem to be solved by the invention]

[0008] When AI (Artificial Intelligence) is used to automatically classify the type of disease detected during an examination of a body cavity using an endoscopic device, the AI ​​performing the automatic classification uses images acquired during the examination as input and outputs the classification results.

[0009] However, images acquired during an examination may not be suitable for classification using AI, such as images in which the lesion is hidden by some object, or images in which the lesion is blurred, even if it is present. In such cases, AI may output inappropriate classification results.

[0010] The inventions described in Patent Documents 1 to 4 may output inappropriate classification results when classifying images that are not suitable for the above-mentioned automatic classification. On the other hand, it is difficult for an observer to determine whether the classification result is appropriate or not.

[0011] The present invention has been made in consideration of the above circumstances, and aims to provide a medical image processing device, an endoscopic system, a medical image processing system, an operating method of a medical image processing device, a program, and a storage medium that enable an observer to understand whether or not a classification made in an automatic classification of medical images is appropriate. [Means for solving the problem]

[0012] In order to achieve the above object, the following aspects of the invention are provided.

[0013] The medical image processing device of the first embodiment includes an image acquisition unit that acquires an image generated by imaging a living body, a classification unit that classifies the image into two or more classes, an image generation unit that generates a region image that visualizes the position of a region that contributed to classification using the classification unit in the image to be classified by the classification unit, and a display signal transmission unit that transmits the image, the classification result derived using the classification unit, and a display signal representing the region image to a display device, and the display signal transmission unit transmits a display signal to the display device to display the region image separately from the image.

[0014] According to the first aspect, a captured image is classified into two or more classes, a region image is generated by visualizing a region in the captured image that contributed to the classification, and the region image is displayed separately from the captured image using a display device, thereby enabling an observer to grasp the region that contributed to the classification and to grasp whether the classification is appropriate.

[0015] The captured image may be a medical image captured using a medical device. The medical image refers to an image of a living body captured using a modality such as an endoscope device, a CT device, an MRI device, or an X-ray imaging device.

[0016] As classes, defined medical classifications may be applied.

[0017] In a second aspect, in the medical image processing apparatus of the first aspect, the image generating section may be configured to change the visualization form of the regional image depending on a classification result of the captured image.

[0018] According to the second aspect, it becomes easier to recognize the distinction between categories.

[0019] Examples of the visualization form include a visualization form using color and a visualization form changing the position. A visualization form using both of these is also possible.

[0020] A third aspect may be configured such that, in the medical image processing device of the first or second aspect, the classification unit classifies the captured image based on features acquired from the captured image, and the image generation unit generates a region image based on the features.

[0021] According to the third aspect, it is possible to generate a regional image based on the feature amount of a captured image.

[0022] In the third aspect, the captured image may be divided into a plurality of regions, a feature amount may be calculated for each region, and a region that contributed to the classification of the captured image may be identified based on the feature of each region.

[0023] In a fourth aspect, in the medical image processing device of the first or second aspect, the classification unit may be configured to apply a trained deep learning device, and the image generation unit may be configured to generate a region image based on information from an intermediate layer in the deep learning device.

[0024] According to the fourth aspect, it becomes possible to generate a region image based on information from the intermediate layer of a trained deep learning device.

[0025] In a fifth aspect, in the medical image processing device of any one of the first to fourth aspects, the classification unit may be configured to calculate a degree of class membership for each of a plurality of areas set in the captured image, and classify the captured image based on the degree of membership.

[0026] According to the fifth aspect, classification based on the degree of belonging is possible.

[0027] Examples of the degree of membership include the probability of membership in a class and the score of a class.

[0028] In a sixth aspect, in the medical image processing apparatus according to the fifth aspect, the image generating section may generate a region image based on the degree of belonging.

[0029] According to the sixth aspect, it is possible to generate a region image based on the degree of belonging.

[0030] In a seventh aspect, in a medical image processing device of any one of the first to sixth aspects, the classification unit may perform an exception determination for the classification based on the area image, and the display signal transmission unit may transmit a display signal representing the result of the exception determination using the classification unit to the display device.

[0031] According to the seventh aspect, it is possible to prevent the output of an inappropriate classification result for a captured image that is difficult to classify.

[0032] The display signal transmission unit may transmit, to the display device, a display signal representing the result of the exception determination, instead of the classification result. The display signal transmission unit may transmit, to the display device, a display signal representing the result of the classification result and the result of the exception determination.

[0033] In an eighth aspect, in the medical image processing device of the seventh aspect, the classification unit may be configured to calculate a reliability of the classification result derived using the classification unit based on the region image, and the display signal transmission unit may be configured to transmit a display signal representing the reliability to the display device.

[0034] According to the eighth aspect, the reliability of the classification can be grasped.

[0035] A ninth aspect is a medical image processing device according to the eighth aspect, further comprising a storage instruction acquisition unit that acquires an instruction to store the captured image, and a storage unit that stores the captured image, and the storage unit may be configured to associate the captured image with at least one of the classification result, the result of the exception determination, and the reliability of the classification result when storing the captured image in the storage unit based on the instruction to store the captured image.

[0036] According to the ninth aspect, the captured image and information associated with the captured image can be used, and the information associated with the captured image can be confirmed.

[0037] A tenth aspect may be configured such that in the medical image processing apparatus according to any one of the first to ninth aspects, the display signal transmission unit transmits a display signal representing text information of the classification result to the display device.

[0038] According to the tenth aspect, the classification result can be understood based on the character information.

[0039] The text information can be in any language. Abbreviations can be used for the text information.

[0040] The medical image processing system of the eleventh aspect includes an image acquisition unit that acquires an image generated by imaging a living body, a classification unit that classifies the image into two or more classes, an image generation unit that generates a region image that visualizes the position of a region that contributed to classification using the classification unit in the image to be classified by the classification unit, a display device that displays the image and the region image, and a display signal transmission unit that transmits a display signal representing the image and the region image to the display device, and the display signal transmission unit transmits a display signal to the display device to display the region image separately from the image.

[0041] According to the eleventh aspect, it is possible to obtain the same effects as those of the first aspect.

[0042] In the eleventh aspect, the same items as those specified in the second to tenth aspects can be appropriately combined. In this case, the components performing the processes or functions specified in the medical image processing device can be understood as the components of the medical image processing system performing the corresponding processes or functions.

[0043] The medical image processing method according to the 12th aspect includes an image acquisition step of acquiring an image generated by imaging a living body, a classification step of classifying the image into two or more classes, an image generation step of generating a region image that visualizes the positions of regions that contributed to the classification in the classification step in the image to be classified in the classification step, and a display signal transmission step of transmitting a display signal representing the image, the classification result derived in the classification step, and the region image to a display device, wherein the display signal transmission step is a medical image processing method in which a display signal that displays the region image separately from the image is transmitted to the display device.

[0044] According to the twelfth aspect, it is possible to obtain the same effects as those of the first aspect.

[0045] In the twelfth aspect, the same matters as those specified in the second to tenth aspects can be appropriately combined. In this case, the components performing the processes or functions specified in the medical image processing device can be understood as the components of the medical image processing method performing the corresponding processes or functions.

[0046] The program of the thirteenth aspect is a program that enables a computer to realize an image acquisition function that acquires an captured image generated by capturing an image of a living body, a classification function that classifies the captured image into two or more classes, an image generation function that generates a region image that visualizes the position of a region that contributed to the classification using the classification function in an captured image that is a classification target of the classification function, and a display signal transmission function that transmits to a display device a display signal representing the captured image, the classification result derived using the classification function, and the region image, and transmits to the display device a display signal that displays the region image separately from the captured image.

[0047] According to the thirteenth aspect, it is possible to obtain the same effects as those of the first aspect.

[0048] In the thirteenth aspect, the same items as those specified in the second to tenth aspects can be appropriately combined. In this case, the components performing the processes or functions specified in the medical image processing device can be understood as the components of the program performing the corresponding processes or functions. Effect of the Invention

[0049] According to the present invention, a captured image is classified into two or more classes, a region image is generated by visualizing the region in the captured image that contributed to the classification, and the region image is displayed separately from the captured image using a display device. This allows an observer to understand the region that contributed to the classification and whether the classification is appropriate. [Brief description of the drawings]

[0050] [Figure 1]FIG. 1 is a diagram showing the overall configuration of an endoscope system including a medical image processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing the hardware configuration of the medical image processing apparatus. [Diagram 3] FIG. 3 is a functional block diagram of the medical image processing apparatus according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of a display screen applied to the first embodiment. [Diagram 5] FIG. 5 is an explanatory diagram of a display screen according to a comparative example. [Figure 6] FIG. 6 is an explanatory diagram of a display screen when the reliability of the classification result is low. [Figure 7] FIG. 7 is an explanatory diagram of a display screen showing another example in which the reliability of the classification result is low. [Figure 8] FIG. 8 is an explanatory diagram of the display screen when multiple lesions are present. [Figure 9] FIG. 9 is an explanatory diagram of a display example of the classification result. [Figure 10] FIG. 10 is an explanatory diagram of an example of displaying the membership probability as a classification result. [Figure 11] FIG. 11 is an explanatory diagram of an example of displaying scores as classification results. [Figure 12] FIG. 12 is a flowchart showing the procedure of the medical image processing method. [Figure 13] FIG. 13 is an explanatory diagram of a display screen according to a modified example of the area image. [Figure 14] FIG. 14 is an explanatory diagram of a display screen according to a first modified example of the classification result. [Figure 15] FIG. 15 is an explanatory diagram of a display screen according to a second modified example of the classification result. [Figure 16] FIG. 16 is an explanatory diagram of a region image to which shading according to the contribution degree is applied. [Figure 17] FIG. 17 is an explanatory diagram of a region image to which a heat map according to the contribution degree is applied. [Figure 18] FIG. 18 is a block diagram showing an example of the configuration of a classification unit to which a convolution neural network is applied. [Figure 19] FIG. 19 is a conceptual diagram of the transition of the feature shape of a convolution neural network. [Figure 20] FIG. 20 is a conceptual diagram of visualization based on information from the hidden layer of a convolutional neural network. [Figure 21] FIG. 21 is an explanatory diagram of the segmentation technique. [Figure 22] FIG. 22 is an explanatory diagram of a display screen showing that detection is impossible. [Diagram 23] FIG. 23 is an explanatory diagram of a display screen showing that a judgment is impossible. [Figure 24] FIG. 24 is a functional block diagram of a medical image processing apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, the same components are given the same reference numerals, and duplicated explanations will be omitted as appropriate.

[0052] [Overall configuration of endoscope system] Fig. 1 is a diagram showing the overall configuration of an endoscope system including a medical image processing device according to an embodiment. The endoscope system 9 shown in Fig. 1 includes an endoscope 10, a light source device 11, a processor device 12, a display device 13, a medical image processing device 14, an input device 15, and a monitor device 16.

[0053] The endoscope 10 is an electronic endoscope. The endoscope 10 is also a flexible endoscope. The endoscope 10 includes an insertion section 20, an operation section 21, and a universal cord 22. The insertion section 20 is inserted into a subject. The insertion section 20 is formed in an elongated shape with a small diameter as a whole.

[0054] The insertion section 20 includes a flexible section 25, a bending section 26, and a tip section 27. The insertion section 20 is configured by continuously connecting the flexible section 25, the bending section 26, and the tip section 27. The flexible section 25 has flexibility from the base end side to the tip side of the insertion section 20. The bending section 26 has a structure that can be bent when the operation section 21 is operated. The tip section 27 has an imaging optical system and an imaging element 28, etc. built in, which are not shown.

[0055] The imaging element 28 may be a CMOS type imaging element or a CCD type imaging element. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. CCD is an abbreviation for Charge Coupled Device.

[0056] An observation window (not shown) is disposed on the tip surface 27a of the tip portion 27. The observation window is an opening formed on the tip surface 27a of the tip portion 27. A cover (not shown) is attached to the observation window. An imaging optical system (not shown) is disposed behind the observation window. Image light of the observed site is incident on the imaging surface of the imaging element 28 via the observation window and the imaging optical system, etc. The imaging element 28 captures the image light of the observed site that is incident on the imaging surface of the imaging element 28, and outputs an imaging signal. The term "imaging" here includes the meaning of converting reflected light from the observed site into an electrical signal.

[0057] The operation unit 21 is connected to the base end side of the insertion section 20. The operation unit 21 includes various operation members operated by the surgeon. Specifically, the operation unit 21 includes two types of bending operation knobs 29. The bending operation knobs 29 are used when bending the bending section 26. The surgeon may also be called a doctor, an operator, an observer, a user, or the like.

[0058] The operation unit 21 includes an air / water supply button 30 and a suction button 31. The air / water supply button 30 is used when the surgeon performs an air / water supply operation. The suction button 31 is used when the surgeon performs a suction operation.

[0059] The operation unit 21 includes a still image capture instruction unit 32 and a treatment tool introduction port 33. The still image capture instruction unit 32 is operated by the surgeon when capturing a still image of the observed site. The treatment tool introduction port 33 is an opening through which a treatment tool is inserted into a treatment tool insertion passage that passes through the inside of the insertion unit 20. The treatment tool insertion passage and the treatment tool are not shown. The still image is shown in FIG. 3 with the reference numeral 39.

[0060] The universal cord 22 is a connection cord that connects the endoscope 10 to the light source device 11. The universal cord 22 includes a light guide 35, a signal cable 36, and a fluid tube (not shown), which are inserted inside the insertion portion 20.

[0061] The tip of the universal cord 22 is provided with a connector 37 a connected to the light source device 11 , and a connector 37 b branched off from the connector 37 a and connected to the processor device 12 .

[0062] When the connector 37a is connected to the light source device 11, the light guide 35 and a fluid tube (not shown) are inserted into the light source device 11. As a result, the necessary illumination light, water, and gas are supplied from the light source device 11 to the endoscope 10 via the light guide 35 and the fluid tube (not shown).

[0063] As a result, illumination light is irradiated toward the observation site from an illumination window (not shown) on the distal end surface 27a of the distal end 27. In addition, in response to pressing of the air / water supply button 30, gas or water is sprayed from an air / water supply nozzle (not shown) on the distal end surface 27a of the distal end 27 toward an observation window (not shown) on the distal end surface 27a.

[0064] When the connector 37b is connected to the processor device 12, the signal cable 36 is electrically connected to the processor device 12. As a result, an image pickup signal of the observation site is output from the imaging element 28 of the endoscope 10 to the processor device 12 via the signal cable 36, and a control signal is output from the processor device 12 to the endoscope 10.

[0065] In this embodiment, the endoscope 10 has been described as being a flexible endoscope, but the endoscope 10 may be any of various electronic endoscopes, such as a rigid endoscope, that are capable of capturing video of an area to be observed.

[0066] The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37a. The illumination light can be white light or light in a specific wavelength band. The illumination light may be a combination of white light and light in a specific wavelength band. The light source device 11 is configured to be able to appropriately select light in a wavelength band according to the observation purpose as the illumination light.

[0067] The white light may be light in a white wavelength band or light in multiple wavelength bands. The specific wavelength band is a band narrower than the white wavelength band. The specific wavelength band light may be light in one type of wavelength band or light in multiple wavelength bands. The specific wavelength band light may be called special light.

[0068] The processor device 12 controls the operation of the endoscope 10 via the connector 37b and the signal cable 36. The processor device 12 also acquires an imaging signal from the imaging element 28 of the endoscope 10 via the connector 37b and the signal cable 36. The processor device 12 acquires the imaging signal output from the endoscope 10 by applying a specified frame rate.

[0069] The processor device 12 generates an endoscopic image, which is an observation image of the observation site, based on the imaging signal acquired from the endoscope 10. The endoscopic image 38 here includes a moving image. The endoscopic image 38 may also include a still image 39. The moving image is indicated by reference numeral 38a and is illustrated in FIG. 3. The endoscopic image 38 shown in the embodiment is an example of a captured image.

[0070] When the still image capturing instruction unit 32 of the operation unit 21 is operated, the processor unit 12 generates a still image 39 of the observed region based on the imaging signal acquired from the imaging element 28 in parallel with the generation of the moving image. The still image 39 may be generated with a higher resolution than the resolution of the moving image.

[0071] When generating the endoscopic image 38, the processor device 12 performs image quality correction by applying digital signal processing such as white balance adjustment and shading correction. The processor device 12 may add supplementary information defined by the DICOM standard to the endoscopic image 38. DICOM is an abbreviation for Digital Imaging and Communications in Medicine.

[0072] The endoscopic image 38 is an in-vivo image captured inside a subject, i.e., inside a living body. When the endoscopic image 38 is an image captured using light in a specific wavelength band, the endoscopic image 38 is a special light image. The processor device 12 outputs the generated endoscopic image 38 to each of the display device 13 and the medical image processing device 14. The processor device 12 may output the endoscopic image 38 to a storage device (not shown) via a network (not shown) in accordance with a communication protocol conforming to the DICOM standard. The network may be the network 140 shown in FIG. 2.

[0073] The display device 13 is connected to the processor device 12. The display device 13 displays an endoscopic image 38 transmitted from the processor device 12. The surgeon can perform operations such as moving the insertion section 20 forward and backward while checking the endoscopic image 38 displayed on the display device 13. When the surgeon detects a lesion or the like in the observed region, the surgeon can operate the still image capturing instruction section 32 to capture a still image of the observed region.

[0074] A computer is used as the medical image processing device 14. A keyboard, a mouse, or the like that can be connected to a computer is used as the input device 15. The input device 15 and the computer may be connected by wire or wirelessly. Various monitors that can be connected to a computer are used as the monitor device 16.

[0075] A diagnosis support device such as a workstation or a server device may be used as the medical image processing device 14. In this case, the input device 15 and the monitor device 16 are provided for each of a plurality of terminals connected to the workstation or the like. Furthermore, a medical care support device that supports the creation of medical reports, etc. may be used as the medical image processing device 14.

[0076] The medical image processing device 14 acquires an endoscopic image 38 and stores the endoscopic image 38. The medical image processing device 14 controls playback of the monitor device 16. Note that the term "image" in this specification includes the meanings of image data such as an electrical signal representing an image and information representing an image. The term "image" in this specification means at least either the image itself or image data.

[0077] The term "storing an image" can be read as "saving an image." The term "storing an image" used here means non-temporary storage of an image. The medical image processing device 14 may include a memory for temporary storage that temporarily stores an image.

[0078] The input device 15 is used to input operation instructions to the medical image processing device 14. The monitor device 16 displays an endoscopic image 38 under the control of the medical image processing device 14. The monitor device 16 may function as a display unit for various information in the medical image processing device 14.

[0079] The medical image processing device 14 can be connected to a storage device (not shown) via a network (not shown) in Fig. 1. The image storage format and communication between devices via the network can be based on the DICOM standard and protocols conforming to the DICOM standard.

[0080] The storage device (not shown) may be a storage that non-temporarily stores data, etc. The storage device may be managed by a server device (not shown) A computer that stores and manages various data may be used as the server device.

[0081] The configuration including the medical image processing apparatus 14 and the monitor device 16 shown in the embodiment is an example of a medical image processing system.

[0082] [Description of medical image processing device according to the first embodiment] [Hardware configuration of medical image processing device] Fig. 2 is a block diagram showing the hardware configuration of a medical image processing device. The medical image processing device 14 shown in Fig. 2 includes a control unit 120, a memory 122, a storage device 124, a network controller 126, a power supply device 128, a display controller 130, an input / output interface 132, and an input controller 134. Note that I / O shown in Fig. 2 represents the input / output interface 132.

[0083] The control unit 120, the memory 122, the storage device 124, the network controller 126, the display controller 130, and the input / output interface 132 are connected via a bus 136 so as to be able to communicate data with each other.

[0084] <Control Unit> The control unit 120 functions as an overall control unit, various calculation units, and a storage control unit of the medical image processing apparatus 14. The control unit 120 executes a program stored in a read only memory (ROM) included in the memory 122.

[0085] The control unit 120 may download a program from an external storage device (not shown) via the network controller 126 and execute the downloaded program. The external storage device may be communicably connected to the medical image processing device 14 via the network 140.

[0086] The control unit 120 uses a RAM (random access memory) provided in the memory 122 as a calculation area and executes various processes in cooperation with various programs, thereby realizing various functions of the medical image processing apparatus 14.

[0087] The control unit 120 controls reading of data from the storage device 124 and writing of data to the storage device 124. The control unit 120 may acquire various data from an external storage device via the network controller 126. The control unit 120 can execute various processes such as calculations using the acquired various data.

[0088] The control unit 120 may include one or more processors. Examples of the processor include a field programmable gate array (FPGA) and a programmable logic device (PLD). The FPGA and the PLD are devices whose circuit configuration can be changed after manufacturing.

[0089] Another example of a processor is an application specific integrated circuit (ASIC), which has a circuit configuration that is specially designed to execute a specific process.

[0090] The control unit 120 can apply two or more processors of the same type. For example, the control unit 120 can use two or more FPGAs or two PLDs. The control unit 120 can apply two or more processors of different types. For example, the control unit 120 can apply one or more FPGAs and one or more ASICs.

[0091] When multiple control units 120 are provided, the multiple control units 120 may be configured using one processor. As an example of configuring multiple control units 120 using one processor, there is a form in which one processor is configured using a combination of one or more CPUs (Central Processing Units) and software, and this processor functions as multiple control units 120. Note that in this specification, software is synonymous with program.

[0092] Another example of configuring multiple control units 120 with one processor is to use a processor that realizes the functions of the entire system including multiple control units 120 with one IC chip. A representative example of a processor that realizes the functions of the entire system including multiple control units 120 with one IC chip is SoC (System On a Chip). Note that IC is an abbreviation for Integrated Circuit.

[0093] In this way, the control unit 120 is configured as a hardware structure using one or more processors of various types.

[0094] Memory The memory 122 includes a ROM (not shown) and a RAM (not shown). The ROM stores various programs executed in the medical image processing device 14. The ROM stores parameters and files used in executing the various programs. The RAM functions as a temporary storage area for data, a work area for the control unit 120, and the like.

[0095] Storage Device The storage device 124 non-temporarily stores various data. The storage device 124 may be externally attached to the medical image processing device 14. A large-capacity semiconductor memory device may be applied instead of or in combination with the storage device 124.

[0096] Network Controller The network controller 126 controls data communication with external devices. The control of data communication may include management of data communication traffic. A known network such as a LAN may be applied to the network 140 connected via the network controller 126.

[0097] <Power supply> A large-capacity power supply such as a UPS (Uninterruptible Power Supply) is applied as the power supply device 128. The power supply device 128 supplies power to each unit of the medical image-processing device 14 when the commercial power supply is cut off due to a power outage or the like.

[0098] Display Controller The display controller 130 functions as a display driver that controls the monitor device 16 based on a command signal sent from the control unit 120 .

[0099] Input / Output Interface The input / output interface 132 communicably connects the medical image processing apparatus 14 to an external device. The input / output interface 132 may use a communication standard such as USB (Universal Serial Bus).

[0100] Input Controller The input controller 134 converts the format of a signal input using the input device 15 into a format suitable for processing by the medical image processing device 14. Information input from the input device 15 via the input controller 134 is transmitted to each unit via the control unit 120.

[0101] 2 is an example, and can be added, deleted, or changed as appropriate. The hardware configuration of the medical image processing device 14 shown in FIG. 2 can also be applied to other embodiments of the first embodiment.

[0102] [Explanation of functional blocks of medical image processing device] 3 is a functional block diagram of the medical image processing device according to the first embodiment. The medical image processing device 14 includes an image acquisition unit 40, an image processing unit 42, a display control unit 44, and a storage unit 46. The image acquisition unit 40 acquires an endoscopic image 38 from the processor device 12. The image acquisition unit 40 stores the endoscopic image 38 in an endoscopic image storage unit 46a.

[0103] The image acquiring unit 40 may acquire the endoscopic image 38 from the processor device 12 via an information storage medium such as a memory card. The image acquiring unit 40 may acquire the endoscopic image 38 via a network 140 shown in FIG.

[0104] That is, the image acquisition section 40 can acquire a moving image 38a composed of time-series frame images 38b. The image acquisition section 40 can acquire a still image 39 when a still image is captured during capturing of the moving image 38a.

[0105] The image processing unit 42 includes a classification unit 48 and a region image generating unit 50. The classification unit 48 automatically classifies lesions from the endoscopic image 38. In this specification, the term classification can be read as differentiation.

[0106] That is, the classification unit 48 may classify the endoscopic images 38 into prescribed classes and derive the classification results. When automatically classifying the endoscopic images 38, the classification unit 48 assigns class information to the frame images 38b constituting the moving image 38a. The classification unit 48 may assign class information to all the frame images 38b, or may assign class information to the frame images 38b every few frames. The classification unit 48 may assign class information to the still images 39. Note that the terms "class" and "class information" can be interpreted as "classification results."

[0107] The classification unit 48 associates the classification result with the frame image 38b and stores it in the classification storage unit 46b. Table 1 below shows examples of classes applied to the classification unit 48.

[0108] [Table 1]

[0109] In addition, NICE in Table 1 above is an abbreviation for NBI International Colorectal Endoscopic Classification. NBI is an abbreviation for Narrow Band Imaging. JNET is an abbreviation for The Japan NBI Expert Team.

[0110] The region image generating unit 50 generates a region image representing a region that contributed to the classification in the endoscopic image 38. The region image generating unit 50 stores the region image in the region image storage unit 46c. The region image is illustrated in FIG.

[0111] The display control unit 44 transmits to the monitor device 16 a display control signal for causing the monitor device 16 to display the endoscopic image 38 and the area image generated using the area image generation unit 50. The display control unit 44 transmits to the monitor device 16 a display control signal for causing the monitor device 16 to display text information representing the classification result of the endoscopic image 38 derived using the classification unit 48. The display control unit 44 shown in the embodiment is an example of a display signal transmission unit.

[0112] The monitor device 16 displays the endoscopic image 38, the area image, and text information representing the classification result on the same screen. The monitor device 16 displays the endoscopic image 38 and the area image in separate areas on the screen. The text information representing the classification result is illustrated in FIG. 4 using the reference numeral 212. Details of the screen displayed by the monitor device 16 will be described later.

[0113] The storage unit 46 includes an endoscopic image storage unit 46a, a classification storage unit 46b, and a region image storage unit 46c. The endoscopic image storage unit 46a stores the endoscopic image 38 acquired by the image acquisition unit 40.

[0114] The image processing unit 42 reads out the endoscopic image 38 stored in the endoscopic image storage unit 46a, and performs image processing on the endoscopic image 38. The display control unit 44 reads out the endoscopic image 38 stored in the endoscopic image storage unit 46a, and causes the monitor device 16 to display the endoscopic image 38.

[0115] The classification storage unit 46b stores the class of the endoscopic image 38 classified by the classification unit 48 in association with the endoscopic image 38. Specifically, the classification storage unit 46b stores the class of the frame images 38b constituting the moving image 38a in association with the frame images 38b. The display control unit 44 reads out the classification result from the classification storage unit 46b, and causes the monitor device 16 to display text information or the like indicating the classification result.

[0116] The region image storage unit 46c stores the region image generated by the region image generation unit 50. The display control unit 44 reads out the region image from the region image storage unit 46c, and causes the monitor device 16 to display the region image.

[0117] The storage unit 46 shown in Fig. 3 may include one or more storage elements. That is, the storage unit 46 may include three storage elements corresponding to the endoscopic image storage unit 46a, the classification storage unit 46b, and the area image storage unit 46c, respectively. Also, the endoscopic image storage unit 46a, the classification storage unit 46b, and the area image storage unit 46c may each include a plurality of storage elements. Furthermore, two or all of the endoscopic image storage unit 46a, the classification storage unit 46b, and the area image storage unit 46c may be configured using a single storage element.

[0118] [Explanation of the display screen displayed on the monitor device] <Explanation of area image> 4 is an explanatory diagram of a display screen applied to the first embodiment. A display screen 200 shown in FIG.

[0119] The endoscopic image display area 202 is an area in which the endoscopic image 38 is displayed. The endoscopic image display area 202 can display a still image 39. The endoscopic image display area 202 can switch between displaying the endoscopic image 38 and the still image 39. Reference numeral 209 denotes a classification contributing area 209 which is an area that contributed to the classification of the endoscopic image 38. Fig. 4 shows a schematic diagram of the classification contributing area 209.

[0120] The region image display region 204 displays a region image 208. A classification contribution corresponding region 210 is highlighted in the region image 208. The classification contribution corresponding region 210 is a region in the region image 208 that corresponds to the classification contribution region 209.

[0121] In the classification result display area 206, text information 212 showing the classification result is displayed. Fig. 4 shows an example in which the English spelling of tumor, "neoplastic," is displayed in the classification result display area 206 as the text information 212. Note that the text information 212 can be in any language. That is, the text information 212 can be in Japanese or in a foreign language other than English. Furthermore, an abbreviation can be used for the text information 212.

[0122] Fig. 5 is an explanatory diagram of a display screen according to a comparative example. The comparison screen 220 displays the results of the automatic classification, and is an example in which a tumor 222 is found in an endoscopic image 38 and the endoscopic image 38 is classified as a tumor. The comparison screen 220 displays the endoscopic image 38 and text information 212 indicating the classification result. Note that the tumor 222 is illustrated diagrammatically in Fig. 5.

[0123] The comparison screen 220 shown in Fig. 5 displays text information 212 indicating the classification result, but if the endoscopic image 38 is difficult to classify, an incorrect classification result may be displayed. On the other hand, the display screen 200 shown in Fig. 4 displays a region image 208 corresponding to the endoscopic image 38, and displays a classification contribution corresponding region 210 in the region image 208. This allows the observer to visually grasp which region of the endoscopic image 38 the classification unit 48 is based on to perform classification. Furthermore, the region image 208 can be an indicator of the reliability of the classification result.

[0124] Fig. 6 is an explanatory diagram of a display screen when the reliability of the classification result is low. The display screen 200a shown in Fig. 6 shows an example of a case where the classification result is incorrect and the classification is made based on an area other than the tumor 222 in the endoscopic image 38.

[0125] In the region image 208a shown in FIG. 6, a region 230 different from the region that should be the classification contribution corresponding region 210a is displayed as the classification contribution corresponding region.

[0126] 7 is an explanatory diagram of a display screen showing another example in which the reliability of the classification result is low. A display screen 200b shown in Fig. 7 is an example in which a tumor 222 cannot be found in the endoscopic image .

[0127] In the region image 208b shown in Fig. 7, the region 210b to be the classification-contributing corresponding region is not displayed. When the display screen 200b shown in Fig. 7 is displayed, it is considered that the classification unit 48 cannot find the classification target due to the fact that the tumor 222 is blurred and the size of the tumor 222 is small.

[0128] In such a case, the operator can operate the endoscope 10 shown in Fig. 1 to adjust the focus or to enlarge and display the lesion and its surroundings, thereby enabling the classification unit 48 to perform correct classification. In other words, when the region image 208 is displayed together with the endoscopic image 38, this can provide the operator with a suggestion of an operation to cause the classification unit 48 to derive a correct classification result.

[0129] In this embodiment, an example is shown in which an endoscopic image display area 202 and a region image display area 204 are displayed on a display screen 200 displayed on one monitor device 16, but it is also possible to display a display screen 200 including an endoscopic image display area 202 on one of two monitor devices and a display screen 200 including a region image display area 204 on the other monitor device.

[0130] Furthermore, the endoscopic image display area 202 and the regional image display area 204 may be switched and displayed in a time-division manner on one display screen 200 displayed on one monitor device 16. Furthermore, the regional image display area 204 may be superimposed on the endoscopic image display area 202. For example, the regional image display area 204 may be superimposed and displayed at a position in the endoscopic image display area 202 that does not interfere with observation of the endoscopic image 38, such as the lower left corner of the endoscopic image display area 202 shown in FIG.

[0131] The observer tends to dislike moving the viewpoint in order to observe the endoscopic image while performing precise operations. When the regional image display area 204 is displayed superimposed on the endoscopic image display area 202, the endoscopic image display area 202 and the regional image display area 204 are disposed closer to each other on the display screen 200b. This has the effect of reducing the observer's viewpoint movement.

[0132] Fig. 8 is an explanatory diagram of a display screen when multiple lesions are present. In a display screen 200d shown in Fig. 8, a first classification contributing region 209a and a second classification contributing region 209b are extracted in correspondence with multiple lesions present in the endoscopic image 38.

[0133] A first classification contribution corresponding region 210g corresponding to the first classification contribution region 209a and a second classification contribution corresponding region 210h corresponding to the second classification contribution region 209b are displayed in the region image 208g. A display mode that allows the first classification contribution corresponding region 210g and the second classification contribution corresponding region 210h to be distinguished from each other is applied.

[0134] In addition, the classification result display area 206a of the display screen 200d displays first character information 212e representing the first classification result corresponding to the first classification contribution corresponding area 210g, and second character information 212f representing the second classification result corresponding to the second classification contribution corresponding area 210h.

[0135] When lesions of different classifications, such as non-neoplastic lesions and neoplastic lesions, exist, it is difficult for a system that outputs one classification result from an endoscopic image 38 to return an appropriate classification result. On the other hand, in the medical image processing device 14 according to the present embodiment, as shown in Fig. 8, a first classification contributing region 209a and a second classification contributing region 209b are visualized using a display mode for each classification in a region image 208g. This makes it possible to obtain an appropriate classification result even when multiple lesions exist in an endoscopic image 38 and the multiple lesions are classified differently.

[0136] <Explanation of classification result display> Fig. 9 is an explanatory diagram of a display example of the classification result. Fig. 9 shows an example in which a specific class is displayed as the classification result. Text information 212a shown in Fig. 9 indicates that the classification result is NICE1.

[0137] Fig. 10 is an explanatory diagram of an example of displaying the membership probability as a classification result. Fig. 10 shows an example of displaying the membership probability of each class as a classification result. The text information 212b shown in Fig. 10 indicates that the membership probability of NICE1 is 98 percent, the membership probability of NICE2 is 2 percent, and the membership probability of NICE3 is 0 percent.

[0138] The text information 212b shown in Fig. 10 may be only text indicating that the attribution probability of NICE1 is 98 percent. The text information 212b shown in Fig. 10 may be text indicating that the attribution probability of NICE1 is 98 percent and that the attribution probability of NICE2 is 2 percent.

[0139] Fig. 11 is an explanatory diagram of an example of displaying scores as classification results. Fig. 11 shows an example of displaying the scores of each class as classification results. Text information 212c shown in Fig. 11 indicates that the score of NICE1 is 1.23, the score of NICE2 is 0.002, and the score of NICE3 is 0.05. The membership probability shown in Fig. 10 and the score shown in Fig. 11 are examples of the degree of class membership.

[0140] [Procedure of medical image processing method] Fig. 12 is a flow chart showing the procedure of the medical image processing method. In an endoscopic image acquiring step S10, the image acquiring unit 40 shown in Fig. 3 acquires an endoscopic image 38. In an endoscopic image storing step S12, the image acquiring unit 40 stores the endoscopic image 38 acquired in the endoscopic image acquiring step S10 in an endoscopic image storing unit 46a.

[0141] In the classification step S14, the classification unit 48 classifies the endoscopic image 38 into a prescribed class. In the classification result storage step S16, the classification unit 48 stores the classification result derived in the classification step S14 in the classification storage unit 46b.

[0142] In a region image generating step S18, the region image generating unit 50 generates a region image such as the region image 208 shown in Fig. 4 based on the classification result. In a region image storing step S20, the region image generating unit 50 stores the region image generated in the region image generating step S18 in the region image storing unit 46c.

[0143] In the display signal transmission step S22, the display control unit 44 transmits a display signal to the monitor device 16. The display signal transmitted from the display control unit 44 to the monitor device 16 includes a display signal representing the endoscopic image 38 and the area image 208. The display signal transmitted from the display control unit 44 to the monitor device 16 may include a display signal representing the classification result.

[0144] [Modification of display screen] <Modification of Area Image> Fig. 13 is an explanatory diagram of a display screen relating to a modified example of the region image. In the display screen 200c shown in Fig. 13, a reduced image of the endoscopic image 38 is synthesized in the background of a region image 208c. That is, in the region image 208c, the classification contribution corresponding region 210 is superimposed on the reduced endoscopic image 38. The reduced image of the endoscopic image 38 may have a lower resolution than the endoscopic image 38.

[0145] <Modification of classification results> Fig. 14 is an explanatory diagram of a display screen according to a first modified example of the classification result. In the region image 208d shown in Fig. 14, text information 212d representing the classification result is superimposed. In the region image 208d, the text information 212d may be highlighted by, for example, changing the color of the text information 212d from the color of the classification contribution corresponding region 210.

[0146] Fig. 15 is an explanatory diagram of a display screen according to a second modified example of the classification result. In the display screen 200e shown in Fig. 15, a color according to the classification result is assigned to the frame 201 of the endoscopic image 38 and the frame 207 of the area image 208h. Note that a color may be assigned only to the frame 201, or only to the frame 207. That is, in the display screen 200e, a color according to the classification result is assigned to at least one of the frame 201 of the endoscopic image 38 and the frame 207 of the area image 208h.

[0147] Since the observer observes the endoscopic image while performing precise operations, there is a concern that the observer's movement of the viewpoint and the observer's visual recognition of text information, etc. may adversely affect the observer's operations. On the other hand, an operator who looks at the display screen 200e shown in Fig. 15 can grasp the classification results with almost no movement of the viewpoint.

[0148] The medical image processing device 14 can not only color the frame 201 etc. according to the classification result, but also color the frame 201 etc. when the classification result is a specific classification, such as when the classification result is a tumor, and not color the frame 201 etc. when the classification result is another classification. In addition, the medical image processing device 14 may color the frame 201 etc. in the case of an exception determination, which will be described later, or may change the color according to the reliability when changing the display according to the reliability.

[0149] The manner of changing the color may include a manner of changing the shade of the same color. For example, if classified as a tumor, the color may be dark red, and if classified as a non-tumor, the color may be light red. The coloring is not limited to the frame 201, etc. Color may be added to areas other than the frame 201 and the frame 207. Furthermore, a display manner other than color, such as a symbol, may be applied as long as it is a manner that suppresses the movement of the operator's viewpoint and makes it easier for the operator to understand the classification result compared to text information.

[0150] Other Modifications The display form of the region image may be changed according to the classification result. For example, when classifying the endoscopic image 38 into two classes, tumor and non-tumor, the color for tumor and the color for non-tumor may be different. In such an example, the color may be red if classified as tumor, and blue if classified as non-tumor.

[0151] Also, the display screen may be configured so that multiple region image display areas 204 can be displayed, and the position of the region image 208 may be changed according to the classification result. For example, in a display screen capable of displaying two region image display areas 204, one above the other, the upper region image display area 204 may be displayed when classified as a tumor, and the lower region image display area 204 may be displayed when classified as a non-tumor. The above-described modified examples of the display screen may assist the operator's visibility. The display form according to the classification result shown in the embodiment is an example of a visualization form.

[0152] Example of displaying area images according to the contribution of classification Fig. 16 is an explanatory diagram of a region image to which shading according to the contribution degree is applied. In a display screen 200e shown in Fig. 16, shading according to the contribution degree is applied to a classification contribution corresponding region 210e of a region image 208e.

[0153] 16, the central vicinity 211a is dark and the periphery 211b is light. This indicates that the contribution of the central vicinity 211a is high and the contribution of the periphery 211b is low. The contribution may be classified into three or more stages.

[0154] Fig. 17 is an explanatory diagram of a region image to which a heat map according to the contribution degree is applied. In a display screen 200f shown in Fig. 17, a heat map according to the contribution degree is applied to a classification contribution corresponding region 210f of a region image 208f.

[0155] 17, red is applied to the central vicinity 211a and blue is applied to the periphery 211b. An intermediate region 211c between the central vicinity 211a and the periphery 211b is applied with an intermediate color that changes from red to blue. Examples of intermediate colors include orange, yellow, and green.

[0156] 17, the contribution of the central vicinity 211a is high, and the contribution decreases from the central vicinity 211a to the periphery 211b, and the contribution of the periphery 211b is the lowest. Note that two or more of the modified examples described here may be appropriately combined.

[0157] Detailed description of classification and visualization of areas that contribute to classification Next, a specific example of classification of the endoscopic image 38 and visualization of areas that contribute to the classification will be described.

[0158] <Pattern 1> As pattern 1, an example is shown in which feature amounts are calculated from the endoscopic image 38, classification is performed based on the feature amounts, and the areas that contribute to the classification are visualized. For classification based on the feature amounts, a method such as SVM (support vector machine) can be applied. For example, a blood vessel area is extracted from the endoscopic image 38, and the feature amounts of the extracted blood vessel area are calculated.

[0159] Other techniques include a technique of performing texture analysis on the endoscopic image 38 and calculating a feature amount using the analysis results, and a technique of calculating a local feature amount such as SIFT (scale-invariant feature transform).

[0160] The feature amount calculated using the above method is analyzed for each region when the target image is divided into multiple regions. This makes it possible to calculate the class membership probability for each region. It becomes possible to perform visualization processing for each region based on the class membership probability for each region. The multiple regions shown in the embodiment are an example of multiple regions set in a captured image.

[0161] <Pattern 2> As Pattern 2, an example is shown in which the information in the intermediate layer of a convolutional neural network is analyzed to classify an endoscopic image 38 and the areas that contribute to the classification are visualized. This method makes it possible to process classification and visualization in parallel.

[0162] 18 is a block diagram showing a configuration example of a classification unit to which a convolutional neural network is applied. Hereinafter, the convolutional neural network may be abbreviated to CNN (Convolutional Neural Network). The classification unit to which the convolutional neural network is applied shown in the embodiment is an example of a deep learning device.

[0163] The classification unit 300 illustrated in Fig. 18 is an example of the classification unit 48 illustrated in Fig. 3. The classification unit 300 includes a CNN 302, an error calculation unit 304, and a parameter update unit 306.

[0164] The CNN 302 performs image recognition of the type of lesion shown in the endoscopic image 38. The CNN 302 has a multiple layer structure and holds multiple weight parameters. The weight parameters of the CNN 302 are updated from the initial values ​​to optimal values, so that an unlearned model can be changed into a trained model.

[0165] The CNN 302 includes an input layer 310, an intermediate layer 312, and an output layer 314. The intermediate layer 312 includes multiple sets of a convolutional layer 320 and a pooling layer 322. The intermediate layer 312 also includes a fully connected layer 324. Each layer has a structure in which multiple nodes are connected by edges.

[0166] An endoscopic image 38, which is the learning target, is input to the input layer 310. The intermediate layer 312 extracts features from the endoscopic image 38 input from the input layer 310. The convolution layer 320 obtains a feature map by applying filtering to nearby nodes in the previous layer. Note that filtering is synonymous with convolution calculation using a filter.

[0167] The pooling layer 322 generates a new feature map by reducing the feature map output from the convolution layer 320. The convolution layer 320 plays a role in extracting features such as edge extraction from the endoscopic image .

[0168] The pooling layer 322 plays a role in providing robustness so that the extracted features are not affected by translation, etc. Note that the intermediate layer 312 is not limited to a set of the convolution layer 320 and the pooling layer 322, but may have a convolution layer 320 that is continuous, or may include a normalization layer (not shown).

[0169] The output layer 314 classifies the type of lesion in the endoscopic image 38 based on the features extracted by applying the intermediate layer 312. The trained CNN 302 may classify medical images into two classes, tumor and non-tumor, for example. The recognition result may be two types of scores corresponding to tumor and non-tumor.

[0170] The filter coefficients and offset values ​​applied to the convolutional layer 320 in the CNN 302 before training, and the weights of the connections to the next layer in the fully connected layer 324 are set to arbitrary initial values.

[0171] The error calculation unit 304 obtains the recognition result output from the output layer 314 of the CNN 302 and the correct answer data 370 for the endoscopic image 38, and calculates the error between them. Examples of the error calculation method include softmax cross entropy and sigmoid.

[0172] The parameter update unit 306 adjusts the weight parameters of the CNN 302 by applying the error backpropagation method based on the error calculated by the error calculation unit 304. The parameter update unit 306 repeatedly performs the parameter adjustment process, and repeats learning until the difference between the output of the CNN 302 and the ground truth data 370 becomes small.

[0173] The classification unit 300 uses a data set of endoscopic images 38 stored in a database (not shown) and performs learning to optimize each parameter of the CNN 302, thereby generating a trained model.

[0174] Fig. 19 is a conceptual diagram of the shape transition of features of a convolutional neural network. Reference numeral 330 indicates schematic information. Input data represents information 330a of the input layer 310. Conv.1 to Conv.5 represent the intermediate layer 312. FC6 to FC8 represent the fully connected layer 324. In Fig. 19, the amount of information 330a of the input layer 310 and the amount of information 330b of the intermediate layer 312 are represented using width x height x number of channels.

[0175] The convolutional neural network performs final output while aggregating spatial information of the input image. As shown in Fig. 19, the convolutional neural network aggregates spatial information having width and height as it progresses from the input layer 310 to the intermediate layer 312, and loses spatial information after information 330c of the fully connected layer 324.

[0176] In other words, it is possible to extract features that retain spatial information by analyzing the information 330b of the intermediate layer 312. However, the information 330b of the intermediate layer 312 generally has a greater number of channels than the information 330a of the input layer 310.

[0177] 19, the number of channels of information 330a in the input layer 310 is 3, whereas the number of channels of information 330b in Conv.4 of the hidden layer 312 increases to 384. Therefore, the problem is how to aggregate information in the channel direction to visualize the classification contribution region. An example of aggregating information in the channel direction and visualizing the classification contribution region is shown below.

[0178] Example 1 In general, channels that contribute significantly to the final output tend to have large absolute values. Therefore, the absolute values ​​of the information in each channel of the hidden layer 312 are compared, and the channel with the larger absolute value is extracted. The extracted channels are visualized. This makes it possible to visualize the areas that contribute more significantly to the final output.

[0179] When extracting a channel, multiple channels may be extracted. As an example of the case where multiple channels are extracted, a specified number of channels may be extracted in descending order of absolute value. When extracting multiple channels, the multiple channels may be averaged.

[0180] 《Second example》 It is also possible to perform principal component analysis in the channel direction of the intermediate layer 312, extract principal components, and visualize the extracted principal components. For example, it is possible to visualize the extracted principal components by reducing the channel dimension to one dimension. In the second example, since information on all channels is visualized, it is possible to visualize more accurately than in the first example.

[0181] 《Third example》 The final output result is a score for each class label in the classification. The contribution of each class label in the classification to the score can be derived using differentiation. For example, Grad-CAM (Gradient-weighted Class Activation Mapping) can be applied to derive the contribution of each class label in the classification to the score.

[0182] Let the score of any class c be y c Let the feature map of the kth channel of any hidden layer be A k Feature map A k The value of the coordinate (i,j) of A k ij Let L be the map that visualizes the contribution of class c. C Grad-CAM is obtained by the following formula:

[0183]

number

[0184] Map L C Grad-CAM shows a region image visualizing the classification contribution region that aggregates channel direction information.

[0185] Fig. 20 is a conceptual diagram of visualization based on information from the intermediate layer of a convolutional neural network. The example shown in Fig. 20 shows an example in which a processing target image 380 containing two types of animals is classified into one animal 381a or the other animal 381b, and the areas that are the basis for the classification are visualized.

[0186] A first classified image 382 shows an example in which an animal is classified as one animal 381a and a region 384 that is the basis for the classification is visualized. A second classified image 386 shows an example in which an animal is classified as the other animal 381b and a region 388 that is the basis for the classification is visualized.

[0187] When applying pattern 2 to classification of endoscopic image 38, the processing target image 380 may be set to endoscopic image 38, and two types of animals may be set to two types of classes. The two types of animals may be set to two types of feature regions.

[0188] <Pattern 3> As pattern 3, an example of visualizing an area by applying a segmentation technique is shown. Fig. 21 is an explanatory diagram of the segmentation technique. Fig. 21 shows an example of classifying an image 390 to be classified, which includes a person 391a and an animal 391b, into a person area 394, an animal area 396, and a background area 398. Reference numeral 392 denotes a classified image.

[0189] In the segmentation technique, a class membership probability for each region is obtained. It is possible to visualize each region based on the membership probability. In addition, it is possible to classify the classification target image 390 based on the region obtained by applying the segmentation technique. For example, the class that occupies the largest area of ​​the classification target image 390 among the classes applied for classification may be the classification result.

[0190] [Effects of the First Embodiment] According to the medical image processing apparatus according to the first embodiment, the following advantageous effects can be obtained.

[0191] [1] An endoscopic image 38 is displayed in an endoscopic image display area 202 on a display screen 200 displayed on the monitor device 16. A region image 208 is displayed in a region image display area 204 separate from the endoscopic image display area 202 on the display screen 200. The region image 208 includes a classification contribution corresponding area 210 corresponding to a classification contribution area 209 that contributed to the classification of the endoscopic image 38. This allows the observer to visually grasp which area of ​​the endoscopic image 38 the classification unit 48 is based on when making the classification. Furthermore, the region image 208 can be an indicator of the reliability of the classification result.

[0192] [2] The classification result is displayed on the monitor device 16. This allows the observer to visually confirm the classification result of the endoscopic image 38.

[0193] [3] As a classification result, the degree of membership of each class is derived. The degree of membership of each class is displayed on the monitor device 16. The degree of membership may be determined by applying the probability of membership of each class and the score of each class. This allows the observer to visually recognize the degree of membership of each class.

[0194] [4] The display form of the region image 208 may be changed depending on the classification result, thereby improving the visibility of the classification result.

[0195] [Description of medical image processing device according to the second embodiment] Next, a medical image processing device according to a second embodiment will be described. In the medical image processing device according to the second embodiment, exception judgment is added to the classification of the endoscopic image 38. That is, the lesion area of ​​the endoscopic image 38 is visualized using the patterns 1, 2, and 3 shown in the first embodiment, and then the region image is analyzed to perform exception judgment.

[0196] For example, when the area of ​​a lesion area detected from endoscopic image 38 is equal to or smaller than a prescribed threshold, it may be deemed undetectable. When multiple lesion areas are detected from endoscopic image 38 and the areas of each of the lesion areas are equal to or larger than a prescribed threshold, it may be deemed undeterminable.

[0197] The medical image processing apparatus according to the second embodiment has the same hardware and functional blocks as the medical image processing apparatus 14 according to the first embodiment. The description of the hardware and functional blocks of the medical image processing apparatus according to the second embodiment will be omitted.

[0198] [Display screen when detection is not possible] Fig. 22 is an explanatory diagram of a display screen showing undetectable. Display screen 400 shown in Fig. 22 displays "undetectable" as text information 412. When text information 412 showing undetectable is displayed, classification contribution corresponding region 410 is not displayed in region image 408. Note that in Fig. 22, lesion region 407 in the case of undetectable is diagrammatically illustrated using a solid line. Lesion region 407 in the case of detectable is diagrammatically illustrated using a two-dot chain line.

[0199] The text information 412 may be in a form in which text information representing the classification result is overwritten with text information representing "undetectable." The display screen 400 may be in a form representing "undetectable" separately from the text information representing the classification result.

[0200] The classification unit 48 shown in Fig. 3 may analyze the region image 408 to quantify the reliability of the classification, and the display control unit 44 may display a numerical value such as a score indicating the reliability of the classification on the display screen 400. Note that reference numeral 402 shown in Fig. 22 indicates an endoscopic image display region. Reference numeral 404 indicates a region image display region. Reference numeral 406 indicates a classification result display region.

[0201] [Display screen when judgment is impossible] Fig. 23 is an explanatory diagram of a display screen showing "undeterminable". In the display screen 400a shown in Fig. 23, "undeterminable" is displayed as text information 412a. When the text information 412a showing "undeterminable" is displayed, a classification contribution corresponding region 410a showing "undeterminable" is displayed in the region image 408a.

[0202] An example of the classification contribution corresponding area 410a that indicates that it is impossible to determine is an example in which two types of display forms corresponding to different classifications are mixed. Figure 23 illustrates a classification contribution corresponding area 410a in which two colors are mixed when changing the color according to the classification. Note that the reference numeral 409 in Figure 23 represents a classification contribution area.

[0203] [Effects of the second embodiment] According to the medical image processing apparatus according to the second embodiment, the following advantageous effects can be obtained.

[0204] [1] When a lesion in the endoscopic image 38 is undetectable or undeterminable, text information 412 indicating that the lesion is undetectable or undeterminable is displayed. This allows the operator to know that the classification is inappropriate.

[0205] [2] The reliability of the classification is calculated based on the region image. The reliability is displayed on the display screen. This allows the user to recognize the reliability of the classification.

[0206] [Description of medical image processing device according to the third embodiment] Fig. 24 is a functional block diagram of a medical image processing device according to the third embodiment. A medical image processing device 14a shown in Fig. 24 includes a storage instruction acquisition unit 41 in addition to the medical image processing device 14 shown in Fig. 3. The storage instruction acquisition unit 41 acquires a storage instruction for the endoscopic image 38 transmitted from the processor device 12 shown in Fig. 1. The storage instruction acquisition unit 41 may acquire a storage instruction for the endoscopic image 38 from the endoscope 10.

[0207] When the storage instruction acquisition unit 41 acquires an instruction to store an endoscopic image 38, the endoscopic image 38 acquired using the image acquisition unit 40 is stored in the endoscopic image memory unit 46a in association with at least one of the classification result, the area image, the exception determination result, and the reliability.

[0208] The medical image processing device 14a may combine the classification result, etc. with the endoscopic image 38 and store it, or may store the classification result, etc. separately from the endoscopic image 38. When the operator operates an operation button, etc. (not shown), an instruction to store the endoscopic image 38 may be transmitted from the processor device 12, or an instruction to store the endoscopic image 38 may be automatically transmitted from the processor device 12 based on the classification result, etc. When the reliability is equal to or greater than a specified threshold, the processor device 12 may determine that the medical image processing device 14a has obtained an appropriate classification result and automatically transmit an instruction to store the endoscopic image 38.

[0209] [Effects of the Third Embodiment] According to the medical image processing apparatus according to the third embodiment, the following advantageous effects can be obtained.

[0210] [1] After the endoscopy, when the user creates a report, the user can check whether the classification results, etc. are appropriate.

[0211] [2] When retesting, the results of previous tests can be referred to.

[0212] [Modification of the endoscope system] [Modifications of the processor device] The processor device 12 may have the function of the medical image processing device 14. That is, the processor device 12 may be configured integrally with the medical image processing device 14. In such an embodiment, the display device 13 may also serve as the monitor device 16. The processor device 12 may include a connection terminal for connecting an input device 15.

[0213] [Modification of illumination light] An example of a medical image that can be acquired using the endoscope system 9 shown in this embodiment is a normal light image obtained by irradiating light in the white band, or light in a plurality of wavelength bands as the white band light.

[0214] Another example of a medical image that can be acquired using the endoscope system 9 shown in this embodiment is an image obtained by irradiating light in a specific wavelength range. The specific wavelength band can be a band narrower than the white band. The following modifications can be applied.

[0215] Example 1 A first example of the particular wavelength band is the blue or green band of the visible range. The first example wavelength band includes a wavelength band from 390 nanometers to 450 nanometers or from 530 nanometers to 550 nanometers, and the first example light has a peak wavelength within the wavelength band from 390 nanometers to 450 nanometers or from 530 nanometers to 550 nanometers.

[0216] <Example 2> A second example of the specific wavelength band is the red band of the visible range, the second example wavelength band includes a wavelength band from 585 nanometers to 615 nanometers or from 610 nanometers to 730 nanometers, and the second example light has a peak wavelength within the wavelength band from 585 nanometers to 615 nanometers or from 610 nanometers to 730 nanometers.

[0217] <Example 3> A third example of the specific wavelength band includes a wavelength band where the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and the light of the third example has a peak wavelength in the wavelength band where the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. The wavelength band of this third example includes a wavelength band of 400±10 nanometers, 440±10 nanometers, 470±10 nanometers, or a wavelength band of 600 nanometers or more and 750 nanometers or less, and the light of the third example has a peak wavelength in the wavelength band of 400±10 nanometers, 440±10 nanometers, 470±10 nanometers, or a wavelength band of 600 nanometers or more and 750 nanometers or less.

[0218] Example 4 A fourth example of a specific wavelength band is a wavelength band of excitation light that is used to observe the fluorescence emitted by fluorescent substances in living organisms and excites the fluorescent substances. For example, a wavelength band of 390 nanometers to 470 nanometers. Note that the observation of fluorescence is sometimes called fluorescence observation.

[0219] <Example 5> A fifth example of the specific wavelength band is an infrared wavelength band, which includes a wavelength band from 790 nanometers to 820 nanometers or from 905 nanometers to 970 nanometers, and the fifth example light has a peak wavelength in the wavelength band from 790 nanometers to 820 nanometers or from 905 nanometers to 970 nanometers.

[0220] [Example of special light image generation] The processor device 12 may generate a special light image having information of a specific wavelength band based on a normal light image captured using white light. Note that the term "generate" includes "acquire." In this case, the processor device 12 functions as a special light image acquirer. The processor device 12 acquires a signal of a specific wavelength band by performing a calculation based on color information of red, green, and blue, or cyan, magenta, and yellow, contained in the normal light image.

[0221] Red, green, and blue are sometimes expressed as RGB (Red, Green, Blue), and cyan, magenta, and yellow are sometimes expressed as CMY (Cyan, Magenta, Yellow).

[0222] [Example of generating feature image] As medical images, feature images can be generated using calculations based on at least one of a normal light image obtained by irradiating white light or light of multiple wavelength bands as white light, and a special light image obtained by irradiating light of a specific wavelength band.

[0223] [Example of application to a program that makes a computer function as an image processing device] The image processing method described above can be configured as a program that causes a computer to realize functions corresponding to each step in the image processing method. For example, a program can be configured to cause a computer to realize an endoscopic image acquisition function, an image processing function, a display signal transmission function, and a storage function. The image processing function can include a classification function and a region image generation function.

[0224] A program for causing a computer to realize the above-mentioned image processing functions can be stored in a computer-readable information storage medium, which is a tangible, non-transitory information storage medium, and the program can be provided through the information storage medium.

[0225] Further, instead of providing the program by storing it on a non-transitory information storage medium, a program signal may be provided via a network.

[0226] [Combinations of embodiments and modified examples] The components described in the above-mentioned embodiment and the components described in the modified examples can be used in appropriate combination, and some of the components can be replaced with each other.

[0227] [Examples of application to other devices] In the above embodiment, an endoscopic image is exemplified as a medical image, but the automatic classification shown in this embodiment can also be applied to medical images acquired using a CT device, an MRI device, an X-ray imaging device, etc. The medical image shown in the embodiment is an example of an imaged image.

[0228] The above-described embodiment of the present invention may be modified, added, or deleted as appropriate within the scope of the gist of the present invention. The present invention is not limited to the above-described embodiment, and many modifications may be made by a person having ordinary knowledge in the relevant field within the technical concept of the present invention. [Explanation of symbols]

[0229] 9 Endoscope System 10 Endoscopy 11 Light source device 12 Processor Unit 13 Display device 14 Medical image processing equipment 14a Medical image processing device 15 Input Devices 16 Monitor Device 20 Insertion section 21 Control section 22 Universal Code 25 Soft part 26 Curved section 27 Tip 27a Tip surface 28 Image sensor 29 Curvature control knob 30 Air / water supply button 31 Suction button 32 Still image capture instruction unit 33 Treatment tool inlet 35 Light Guide 36 Signal Cable 37a Connector 37b Connector 38 Endoscopic Images 38a Video 38b Frame image 39 Still Images 40 Image acquisition unit 41 Memory instruction acquisition unit 42 Image processing section 44 Display control section 46 Memory section 46a Endoscope image storage unit 46b Classification storage section 46c Area image storage unit 48 Classification Department 50 Area image generation unit 120 Control section 122 Memory 124 Storage Device 126 Network Controller 128 Power Supply 130 Display Controller 132 Input / Output Interface 134 Input Controller 136 Bus 140 Network 200 display screen 200a display screen 200b display screen 200c display screen 200d display screen 200e display screen 201 slots 202 Endoscopic image display area 204 Area Image display area 206 Classification result display area 206a Classification result display area 207 slots 208 Area Images 208a Area image 208b area image 208c area image 208d area image 208e Area Image 208f area image 208g Area Image 208h area image 209 Classification Contributing Region 209a First Class Contributing Area 209b Second Class Contributing Area 210 Classification Contribution Corresponding Region 210a Areas to be considered as classification contributing corresponding areas 210b Areas to be considered as classification contributing corresponding areas 210e Classification Contribution Corresponding Region 210f Classification contribution corresponding area 210g First Class Contribution Corresponding Area 210h Second Classification Contribution Corresponding Area 211a Near the center 211b Surroundings 211c intermediate area 212 Textual information 212a Text information 212b Text information 212c Text information 212d Text information 212e First character information 212f Second character information 213a First Lesion 213b Secondary Lesion 220 Comparison screen 222 Tumor 230 areas 300 Classification Department 302 CNN 304 Error calculation section 306 Parameter Update Section 310 Input Layer 312 Middle Class 314 Output Layer 320 Convolutional Layer 322 Pooling Layer 324 fully connected layer 330 Information 330a Input layer information 330b Middle Class Information 330c Fully connected layer information 370 Correct Data 380 Images to be processed 381a One Animal 381b The other animal 382 First Classified Images 384 Areas on which classification is based 386 Second Classified Images 388 Areas on which classification is based 390 Images to be Classified 391a person 391b Animals 392 Classified Images 394 People area 396 Animal area 400 display screen 400a Display Screen 402 Endoscopic image display area 404 Area Image display area 406 Classification result display area 408 Area Image 410 Classification Contribution Corresponding Region 412 Text information 412a Text information S10 to S22: Each step of the medical image processing method

Claims

1. an image acquisition unit that acquires an endoscopic image generated by capturing an image of a living body; a classification unit that classifies a lesion area included in the endoscopic image into two or more classes representing types of lesions, including a type indicating a first lesion and a type indicating a second lesion different in type from the first lesion; a region image generating unit that generates a region image in which a classification contribution corresponding region corresponding to a position of a classification contribution region that contributed to the classification of a lesion region included in the endoscopic image is visualized, the region image generating unit visually displays the classification contribution corresponding region in the region image in a different display form depending on the classification result of the classifying unit for the position corresponding to the classification contribution region; a display control unit that causes a display device to display the region image together with the endoscopic image and displays the region image at a position different from that of the endoscopic image; A medical image processing device comprising:

2. the region image generating unit generates the region image displaying the classification contribution corresponding region in a first display form when the classification unit classifies the lesion region into a first class representing the first lesion; The medical image processing device according to claim 1 .

3. When the classification unit classifies the lesion region into a second class representing the second lesion, the region image generation unit generates the region image displaying the classification contribution corresponding region in a second display form. The medical image processing device according to claim 1 .

4. when the classification unit classifies the two or more lesion regions into two or more classes including a first class representing the first lesion and a second class representing the second lesion, the region image generation unit generates the region image in such a way that the classification contribution corresponding region that contributed to the classification of the first class is displayed in a first display form and the classification contribution corresponding region that contributed to the classification of the second class is displayed in a second display form different from the first display form. The medical image processing device according to claim 1 .

5. The region image generating unit visualizes and displays the classification contribution corresponding region in the region image in a different display form according to the classification result of the classification unit for a position corresponding to the classification contribution region, and displays the classification contribution corresponding region in a heat map according to the contribution degree to the classification result. The medical image processing device according to claim 1 .

6. The area image generating unit visualizes and displays the classification contribution corresponding area in the area image in a different display form according to the classification result of the classification unit at a position corresponding to the classification contribution area, and displays the classification contribution corresponding area by changing at least one of the shading and color according to the contribution degree to the classification result. The medical image processing device according to claim 1 .

7. the classification unit classifies a lesion area included in the endoscopic image into a neoplastic lesion and a non-neoplastic lesion; one of the first lesion and the second lesion is the neoplastic lesion and the other is the non-neoplastic lesion; The medical image processing device according to claim 1 .

8. The classification unit classifies the endoscopic image based on a feature amount acquired from the endoscopic image, The medical image processing apparatus according to claim 1 , wherein the region image generating unit generates the region image based on the feature amount.

9. The classification unit is applied with a trained deep learning device, The medical image processing device according to claim 1 , wherein the region image generating unit generates the region image based on information of an intermediate layer in the deep learning device.

10. The medical image processing device according to claim 1 , wherein the classification unit calculates a degree of belonging to the class for each of a plurality of regions set in the endoscopic image, and classifies the endoscopic image based on the degree of belonging.

11. The medical image processing apparatus according to claim 10 , wherein the region image generating unit generates the region image based on the degree of belonging.

12. The classification unit performs an exception determination for the classification based on the region image, The medical image processing apparatus according to claim 1 , wherein the display control unit causes the display device to perform a display showing a result of the exception determination made by the classification unit.

13. The classification unit calculates a reliability of a classification result derived using the classification unit based on a region image; The medical image processing apparatus according to claim 12 , wherein the display control unit causes the display device to perform a display indicating the reliability.

14. a storage instruction acquisition unit that acquires an instruction to store the endoscopic image; A storage unit that stores the endoscopic image; Equipped with The medical image processing device according to claim 13, wherein the memory unit associates at least one of the classification result, the result of the exception determination, and the reliability of the classification result with the endoscopic image when storing the endoscopic image in the memory unit based on an instruction to store the endoscopic image.

15. The medical image processing apparatus according to claim 1 , wherein the display control unit causes the display device to perform a display showing text information of the classification result derived by the classification unit.

16. The endoscopic image is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band. The medical image processing device according to claim 1 .

17. The endoscopic image is a special light image obtained by irradiating light of a specific wavelength band. The medical image processing device according to claim 1 .

18. An endoscope and a light source device for supplying illumination light to the endoscope; A medical image processing device according to any one of claims 1 to 17, The image acquisition unit acquires the endoscopic image from the endoscope. Endoscopy system.

19. an image acquisition unit that acquires an endoscopic image generated by capturing an image of a living body; a classification unit that classifies a lesion area included in the endoscopic image into two or more classes representing types of lesions, including a type indicating a first lesion and a type indicating a second lesion different in type from the first lesion; a region image generating unit that generates a region image in which a classification contribution corresponding region corresponding to a position of a classification contribution region that contributed to the classification of a lesion region included in the endoscopic image is visualized, the region image generating unit visually displays the classification contribution corresponding region in the region image in a different display form depending on the classification result of the classifying unit for the position corresponding to the classification contribution region; a display device for displaying the endoscopic image and the area image; a display control unit that causes the display device to display the region image together with the endoscopic image and displays the region image at a position different from that of the endoscopic image; A medical image processing system comprising:

20. A method for operating a medical image processing device, the medical image processing device including an image acquisition unit, a classification unit, a region image generation unit, and a display control unit; an image acquiring step in which the image acquiring unit acquires an endoscopic image generated by capturing an image of a living body; a classification step in which the classification unit classifies a lesion area included in the endoscopic image into two or more classes representing types of lesions, including a type indicating a first lesion and a type indicating a second lesion different in type from the first lesion; a region image generating step in which the region image generating unit generates a region image in which a classification contribution corresponding region corresponding to a position of a classification contribution region that contributed to the classification of a lesion region included in the endoscopic image is visualized, the region image generating step being for visualizing and displaying the classification contribution corresponding region in the region image in a different display form depending on the classification result of the classifying unit for a position corresponding to the classification contribution region; a display step in which the display control unit causes a display device to display the region image together with the endoscopic image and displays the region image at a position different from that of the endoscopic image; A method of operating a medical imaging device comprising:

21. Computer, an image acquisition unit that acquires an endoscopic image generated by capturing an image of a living body; a classification unit that classifies a lesion area included in the endoscopic image into two or more classes representing types of lesions, including a type indicating a first lesion and a type indicating a second lesion different in type from the first lesion; a region image generating unit that generates a region image in which a classification contribution corresponding region corresponding to a position of a classification contribution region that contributed to the classification of a lesion region included in the endoscopic image is visualized, the region image generating unit visually displays the classification contribution corresponding region in the region image in a different display form depending on the classification result of the classifying unit for the position corresponding to the classification contribution region; A program that functions as a display control unit that causes a display device to display the area image together with the endoscopic image and displays the area image at a position different from the endoscopic image.

22. A non-transitory computer-readable storage medium having instructions stored thereon that, when read by a computer, an image acquisition function for acquiring an endoscopic image generated by capturing an image of a living body; a classification function for classifying a lesion area included in the endoscopic image into two or more classes representing types of lesions, including a type representing a first lesion and a type representing a second lesion different in type from the first lesion; a region image generating function for generating a region image in which a classification contribution corresponding region corresponding to a position of a classification contribution region that contributed to the classification of a lesion region included in the endoscopic image is visualized, the region image generating function visualizing and displaying the classification contribution corresponding region in the region image in a different display form depending on the classification result of the classification function for the position corresponding to the classification contribution region; and A storage medium that causes a computer to execute a display control function of displaying the area image on a display device together with the endoscopic image and displaying the area image at a position different from that of the endoscopic image.

Citation Information

Patent Citations

  • Method of and device for forming fluorescent face of color picture tube

    JP1980028255A

  • Improved type accelerator and brake

    JP1980076711A

  • Picture display

    JP2008036028A

  • Endoscope system, processor for endoscope, and endoscopy aiding method

    JP2010172673A

  • Image processing apparatus and image processing method

    JP2017070609A