Medical image processing device, endoscope system, medical image processing method and recording medium

Abstract

The object of the present invention is to provide a medical image processing device, an endoscope system, a medical image processing method and a program that assist a user in accurately estimating the size of a region of interest. The medical image processing device involved in one embodiment of the present invention is a medical image processing device having a processor, wherein the processor acquires time-series images, determines whether a region of interest in the image is suitable for size estimation, and notifies the judgment result and operation assistance information for improving the judgment result through a notification device.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, an endoscope system, a medical image processing method and a recording medium. Background Art

[0002] In the field of inspection or diagnosis using medical images, a system is expected to assist users in more efficiently performing inspections, etc. For example, Patent Document 1 describes a method of providing satisfactory images from image data by discarding low-quality image frames, thereby standardizing the discovery and analysis of polyps, etc.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application No. 2010-512173 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] Among the above-mentioned user assistance, the size estimation assistance of the target object (area of ​​interest) such as lesions can be flexibly used to determine the excision of the lesion, save the labor and time of preparing the postoperative report, etc. However, depending on the equipment using medical images such as endoscopes, there are problems such as image distortion caused by the optical system, and sometimes the equipment such as the observer is difficult to operate and cannot properly photograph the target object. Therefore, it is sometimes difficult to accurately estimate the size of the target object, and it is impossible to keep the estimation result consistent.

[0008] In addition, although endoscopes can perform inspections, diagnosis, and treatments at the same time, if the parts believed to be lesions are treated blindly, it will be a great burden on users and patients. Therefore, the size of the lesion is used as an indicator of treatment, and the user determines whether to perform resection, which forceps to use, etc. based on this. This is because as the size of the lesion increases, the possibility of malignant transformation increases. However, it is usually difficult to accurately and immediately measure the size of the lesion. This is because the size measurement is greatly affected by the difficulty of operating the observer, the aberration of the lens, the shape change (expansion) of the large intestine, etc., and endoscopic examinations that must be inspected, diagnosed, and treated in real time require very high technical capabilities and concentration.

[0009] However, the prior art such as Patent Document 1 does not take this situation into consideration.

[0010] The present invention has been made in view of such circumstances, and an object of the present invention is to provide a medical image processing apparatus, an endoscope system, a medical image processing method, and a program that assist a user in accurately estimating the size of a region of interest.

[0011] Means for solving technical problems

[0012] The medical image processing device involved in the first embodiment of the present invention is a medical image processing device having a processor, wherein the processor acquires time-series images, determines whether a region of interest in the image is suitable for size estimation, and notifies the result of the determination and operation auxiliary information for improving the result of the determination through a notification device. In the first embodiment, it is determined whether a region of interest (object) is suitable for size estimation, and the result of the determination and operation auxiliary information for improving the result of the determination are notified through a notification device. Thus, the result of the determination is improved by the user performing an operation according to the operation auxiliary information, so that the user can be assisted in making an accurate size estimation of the region of interest.

[0013] In the medical image processing apparatus according to the second aspect, in the first aspect, the processor calculates the accuracy of the estimated size by image processing, and determines whether the region of interest is suitable for the estimated size by comparing the accuracy with a preset threshold.

[0014] In the medical image processing apparatus according to the third aspect, in the second aspect, the processor notifies the accuracy through the notifying means.

[0015] A medical image processing apparatus according to a fourth aspect is configured such that, in the second or third aspect, the processor calculates the accuracy based on the sharpness of a region including the region of interest.

[0016] The medical image processing apparatus according to a fifth aspect is characterized in that in any one of the second to fourth aspects, the processor calculates the accuracy based on the distance between a reference position and the region of interest in the acquired image. The reference position may be, for example, the center of the image.

[0017] A medical image processing apparatus according to a sixth aspect is characterized in that, in any one of the second to fifth aspects, the processor calculates the accuracy based on an imaging angle when imaging the region of interest.

[0018] A medical image processing apparatus according to a seventh aspect is provided in any one of the first to sixth aspects, wherein the apparatus comprises a storage device storing reference images representing an estimated imaging state of a fit size, and the processor causes the notification device to display at least one reference image.

[0019] The medical image processing apparatus according to an eighth aspect is characterized in that in any one of the first to seventh aspects, the processor causes the notification device to display the determination result and the operation support information in a first area in the acquired image and / or a second area that does not overlap with the acquired image.

[0020] The medical image processing apparatus according to a ninth aspect is characterized in that, in any one of the first to eighth aspects, the processor determines one target for size estimation based on an image having two or more regions of interest.

[0021] The medical image processing apparatus according to a tenth aspect is characterized in that, in the ninth aspect, the processor determines the region of interest having the largest area among the two or more regions of interest as the target.

[0022] In the medical image processing apparatus according to an eleventh aspect, in the ninth aspect, the processor determines as the target a region of interest having a size estimated with the highest accuracy among the two or more regions of interest.

[0023] In the medical image processing apparatus according to a twelfth aspect, in any one of the first to eleventh aspects, the processor estimates the size of the region of interest, and notifies the estimation result through the notifying device.

[0024] In the medical image processing apparatus according to the thirteenth aspect, in any one of the first to twelfth aspects, the processor recognizes the user's operation, determines whether the recognized user's operation complies with the operation support information, and notifies the determination result through the notification device.

[0025] The endoscope system according to the fourteenth aspect of the present invention comprises: the medical image processing device according to any one of the first to thirteenth aspects; a notification device; and an endoscope which is inserted into a subject as a photographed object to capture a medical image, and the processor acquires the image captured by the endoscope. Since the endoscope system according to the fourteenth aspect comprises the medical image processing device according to any one of the first to thirteenth aspects, it can assist the user in accurately estimating the size of the region of interest.

[0026] In the endoscope system involved in the fifteenth embodiment, in the fourteenth embodiment, the notification device has a display for displaying information on the screen and / or a speaker for outputting sound, and the processor uses at least one of graphics, text, and sound to notify the judgment result and operation assistance information through the display and / or speaker.

[0027] In the endoscope system involved in the sixteenth embodiment, in the fourteenth or fifteenth embodiment, the operation assistance information includes at least one of the following information: first information indicating the direction and / or the amount of movement of the endoscope viewer to be moved; second information indicating the on / off of the air supply and / or water supply from the endoscope viewer; third information indicating the on / off of the lighting from the endoscope viewer and / or the degree of the lighting; fourth information indicating the on / off of image processing for the acquired image and / or the degree of image processing; and fifth information indicating whether a treatment instrument should be used.

[0028] In the endoscope system according to a seventeenth aspect, in any one of the fourteenth to sixteenth aspects, the processor acquires individual information of the endoscope, and performs determination and / or notification based on the individual information.

[0029] The medical image processing method involved in the eighteenth embodiment of the present invention comprises: an image acquisition step for acquiring time series images; an estimated state judgment step for judging whether the area of ​​interest in the image is suitable for the estimated size; and a notification step for notifying the judgment result and operation auxiliary information for improving the judgment result through a notification device.

[0030] According to the medical image processing method involved in the eighteenth embodiment, the user can be assisted to accurately estimate the size of the region of interest in the same manner as in the first embodiment. In addition, the medical image processing method involved in the eighteenth embodiment can also have the same structure as the second to thirteenth embodiments (processes corresponding to the functions performed by the processor). In addition, the medical image processing methods of these embodiments can also be understood as the operation methods of the medical image processing device having a processor.

[0031] The program according to the nineteenth aspect causes a computer to execute the medical image processing method according to the eighteenth aspect. Another aspect of the present invention is a non-transitory recording medium recording a computer-readable code of the program according to the nineteenth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a diagram showing the configuration of an endoscope system according to the first embodiment.

[0033] Figure 2 This is another diagram showing the structure of the endoscope system.

[0034] Figure 3 This is a functional block diagram of the image processing unit.

[0035] Figure 4 It is a diagram showing information recorded in the recording unit.

[0036] Figure 5 This is a flowchart showing the steps of the medical image processing method according to the first embodiment.

[0037] Figure 6 This is a diagram showing an example of a setting screen for notification conditions and notification method.

[0038] Figure 7 It is a diagram showing an example of a setting screen for operation support information.

[0039] Figure 8 This is a diagram showing an example of a state in which the region of interest is not captured clearly.

[0040] Fig. 9 : is a diagram showing the influence of the position of the target area on the size estimation.

[0041] Fig.10FIG. 1 is a diagram showing an example of determining a shape based on the circularity of a target region.

[0042] Fig.11 It is a diagram showing an example of notification of operation support information.

[0043] Fig.12 This is another diagram showing an example of notification on an observation image.

[0044] Fig.13 A diagram showing a display example of a reference image.

[0045] Fig.14 This is a diagram showing an example of accuracy notification.

[0046] Fig.15 This is a diagram showing an example of combining size information and operation support information for notification.

[0047] Fig.16 This is a flowchart showing the process of recognizing a user operation and notifying the result of determination.

[0048] Fig.17 FIG. 1 is a diagram showing an example of notification on an observation image. DETAILED DESCRIPTION

[0049] Hereinafter, embodiments of a medical image processing apparatus, an endoscope system, a medical image processing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings.

[0050] <First Embodiment>

[0051] <Structure of endoscope system>

[0052] Figure 1 is an external view of an endoscope system 10 (endoscope system, medical image processing device). Figure 2 1 is a block diagram showing the main structure of the endoscope system 10. Figure 1 , 2 As shown, the endoscope system 10 is composed of an endoscope scope 100 (image acquisition unit, endoscope scope), a processor 200 (medical image processing device, processor, image acquisition unit, size estimation unit, estimation state determination unit, notification control unit, scope information acquisition unit, operation recognition unit, recording control unit, display control unit), a light source device 300 (light source device), and a monitor 400 (display device, notification device, display).

[0053] <Structure of endoscope>

[0054] The endoscope 100 includes a hand operation unit 102 and an insertion unit 104 connected to the hand operation unit 102. The operator (user) holds the hand operation unit 102 to operate and inserts the insertion unit 104 into the body of the subject (living body) for observation. In addition, the hand operation unit 102 is provided with an air and water supply button 141, a suction button 142, a function button 143 to which various functions are assigned, and a shooting button 144 for receiving shooting instruction operations (still images, moving images).

[0055] The hand operation unit 102 is provided with an endoscope information recording unit 139 for recording individual information (individual information, endoscope information) of the endoscope 100. The individual information includes, for example, the type of the endoscope 100 (direct view or side view, etc.), the model, the individual identification number, the characteristics of the optical system (angle of view, distortion, etc.), etc. The endoscope information acquisition unit 230 (endoscope information acquisition unit, individual information acquisition unit; see Figure 3 ) acquires the individual information and uses it for processing by the estimated state determination unit 226 and the notification control unit 228. In addition, the observer information recording unit 139 may also be provided in the optical guide connector 108.

[0056] The insertion portion 104 is composed of a soft portion 112, a bending portion 114, and a top hard portion 116 in order from the hand operation portion 102 side. That is, the bending portion 114 is connected to the base end side of the top hard portion 116, and the soft portion 112 is connected to the base end side of the bending portion 114. The hand operation portion 102 is connected to the base end side of the insertion portion 104. The user can bend the bending portion 114 by operating the hand operation portion 102 to change the direction of the top hard portion 116 upward, downward, leftward, and rightward. The top hard portion 116 is provided with a photographic optical system 130, an illumination portion 123, a forceps opening 126, etc. (refer to Figure 1 , 2 ).

[0057] During observation and treatment, the operation unit 208 (see Figure 2 ) can be operated to irradiate white light and / or narrow-band light (one or more of red narrow-band light, green narrow-band light, blue narrow-band light, and purple narrow-band light) from the illumination lenses 123A and 123B of the illumination unit 123. In addition, the air and water supply button 141 can be operated to discharge cleaning water from a water supply nozzle (not shown) to clean the photographic lens 132 (photographic lens, photographic unit) and the illumination lenses 123A and 123B of the photographic optical system 130. A pipeline (not shown) is connected to the forceps opening 126 opened in the top hard part 116, and a treatment instrument (not shown) for removing a tumor, etc., is inserted into the pipeline, and can be appropriately inserted and removed to perform necessary treatment on the subject.

[0058] like Figure 1 ,2 As shown, a photographic lens 132 (image acquisition unit) is disposed on the tip side end surface 116A of the tip hard portion 116. A CMOS (Complementary Metal-Oxide Semiconductor) type imaging element 134 (imaging element, image acquisition unit), a driving circuit 136 (image acquisition unit), and an AFE 138 (AFE: Analog Front End, image acquisition unit) are disposed on the back side of the photographic lens 132, and image signals are outputted using these components. The imaging element 134 is a color imaging element, and has a plurality of pixels composed of a plurality of light receiving elements arranged in a matrix (two-dimensional arrangement) by a specific pattern arrangement (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.). Each pixel of the imaging element 134 includes a microlens, a red (R), green (G) or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographic optical system 130 can generate a color image based on pixel signals of three colors, red, green, and blue, or can generate an image based on pixel signals of any one color or two colors among red, green, and blue. In addition, the imaging element 134 can also be a CCD (Charge Coupled Device) type. In addition, each pixel of the imaging element 134 can also have a purple color filter corresponding to the purple light source 310V and / or an infrared filter corresponding to the infrared light source.

[0059] The optical image of the subject is formed on the light receiving surface (imaging surface) of the imaging element 134 by the imaging lens 132 and converted into an electrical signal, which is output to the processor 200 via a signal cable (not shown) and converted into an image signal. Thus, an endoscopic image (image, medical image) of the subject is displayed on the monitor 400 connected to the processor 200.

[0060] In addition, on the distal end side end surface 116A of the distal end hard portion 116, illumination lenses 123A and 123B of the illumination portion 123 are provided adjacent to the photographic lens 132. On the inner side of the illumination lenses 123A and 123B, an emission end of a light guide 170 described later is provided. The light guide 170 is inserted through the insertion portion 104, the hand operation portion 102, and the universal cable 106, and an incident end of the light guide 170 is disposed in the light guide connector 108.

[0061] The user inserts or removes the endoscope 100 (insertion section 104) configured as above into or from a living body as a subject while performing imaging at a certain frame rate (which can be performed under the control of the image acquisition section 220), thereby sequentially capturing time-series images of the living body.

[0062] <Structure of light source device>

[0063] like Figure 2 As shown, the light source device 300 is composed of a light source 310 for illumination, an aperture 330, a condenser lens 340, and a light source control unit 350, and allows observation light to enter the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a purple light source 310V, which respectively emit red, green, blue, and purple narrow-band lights, and can emit red, green, blue, and purple narrow-band lights. The illumination of the observation light of the light source 310 is controlled by the light source control unit 350, and the illumination of the observation light can be changed (increased or reduced) or the illumination can be stopped as needed.

[0064] The light source 310 can emit red, green, blue, and purple narrow-band lights in any combination. For example, it can emit red, green, blue, and purple narrow-band lights at the same time and irradiate white light (ordinary light) as observation light, or it can emit any one or two lights and irradiate narrow-band light (special light). The light source 310 can also include an infrared light source that irradiates infrared light (an example of narrow-band light). In addition, it is also possible to use a light source that irradiates white light and a filter that transmits white light and each narrow-band light to irradiate white light or narrow-band light as observation light.

[0065] <Wavelength band of light source>

[0066] The light source 310 may be a light source that generates a white band, or a light source that generates a plurality of wavelength bands as the white band, or a light source that generates a specific wavelength band narrower than the white wavelength band. The specific wavelength band may be a blue band or a green band in the visible range, or a red band in the visible range. When the specific wavelength band is a blue band or a green band in the visible range, it may include a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and a peak wavelength may be present in the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm. In addition, when the specific wavelength band is a red band in the visible range, it may include a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and a peak wavelength may be present in the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0067] The specific wavelength band may include a wavelength band in which the absorption coefficients of oxidized hemoglobin and reduced hemoglobin are different, and the light of the specific wavelength band has a peak wavelength in the wavelength band in which the absorption coefficients of oxidized hemoglobin and reduced hemoglobin are different. In this case, the specific wavelength band may include a wavelength band of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light of the specific wavelength band has a peak wavelength in 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0068] In addition, the wavelength band of the light generated by the light source 310 may also include a wavelength band of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength band of 790nm to 820nm or 905nm to 970nm.

[0069] In addition, the light source 310 may also be provided with a light source for irradiating excitation light with a peak value of 390 nm or more and 470 nm or less. In this case, a medical image (medical image, in vivo image) having information on fluorescence emitted by a fluorescent substance in a subject (organism) can be obtained. When obtaining a fluorescent image, a fluorescent pigment (fluorescein, acridine orange, etc.) may also be used.

[0070] The light source type (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode) etc.), wavelength, presence or absence of a filter etc. of the light source 310 are preferably configured according to the type of the subject, the purpose of observation etc., and in observation, it is preferred to combine and / or switch the wavelength of the observation light according to the type, part, purpose of observation etc. of the subject. When switching the wavelength, for example, the wavelength of the irradiated light can be switched by rotating a disc-shaped filter (rotating color filter) which is arranged in front of the light source and has a filter that transmits or shields light of a specific wavelength.

[0071] In addition, the imaging element used in the implementation of the present invention is not limited to a color imaging element in which a color filter is provided for each pixel like the imaging element 134, but may also be a monochrome imaging element. When using a monochrome imaging element, the wavelength of the observation light may be switched in sequence to perform imaging in a plane order (color order). For example, the wavelength of the emitted observation light may be switched in sequence between narrow-band lights (purple, blue, green, red), or broadband light (white light) may be irradiated and the wavelength of the emitted observation light may be switched by rotating color filters (red, green, blue, purple, etc.). In addition, one or more narrow-band lights (green, blue, purple, etc.) may be irradiated and the wavelength of the emitted observation light may be switched by rotating color filters (green, blue, purple, etc.). The narrow-band light may be infrared light of two or more wavelengths with different wavelengths (first narrow-band light, second narrow-band light).

[0072] By connecting the optical connector 108 (refer to Figure 1 , 2 ) is connected to the light source device 300, and the observation light irradiated from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and is irradiated toward the observation range from the illumination lenses 123A and 123B.

[0073] <Processor Structure>

[0074] based on Figure 2 The structure of the processor 200 is described below. The processor 200 inputs the image signal output from the endoscope observer 100 via the image input controller 202, performs necessary image processing in the image processing unit 204 (medical image processing unit, processor), and outputs it via the video output unit 206. As a result, the observation image (in vivo image) is displayed on the monitor 400 (display device). These processes are performed under the control of the CPU 210 (CPU: Central Processing Unit, processor). The communication control unit 205 performs communication control regarding the acquisition of medical images, etc. with the hospital system (HIS: Hospital Information System) or the hospital LAN (Local Area Network) not shown in the figure, and / or the external system or network.

[0075] <Function of Image Processing Unit>

[0076] Figure 32 is a functional block diagram of the image processing unit 204. The image processing unit 204 includes an image acquisition unit 220 (image acquisition unit, medical image acquisition unit), a region of interest selection unit 222 (region of interest selection unit), a size estimation unit 224 (size estimation unit), an estimated state determination unit 226 (estimated state determination unit), a notification control unit 228 (notification control unit), an observer information acquisition unit 230 (observer information acquisition unit, individual information acquisition unit), an operation recognition unit 232 (operation recognition unit), a recording control unit 234 (recording control unit), and a display control unit 236 (display control unit). The details of the processing using these functions will be described later.

[0077] The image processing unit 204 can use the above functions to calculate the characteristic amount of the medical image, to emphasize or reduce the components of a specific frequency band, and to emphasize or make a specific object (region of interest, blood vessels at a desired depth, etc.) less obvious. The image processing unit 204 can also include a special light image acquisition unit that acquires a special light image having information on a specific wavelength band based on an ordinary light image obtained by irradiating light of a white band or irradiating light of multiple wavelength bands as the white band. In this case, the signal of the specific wavelength band can be obtained by calculation based on the color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) contained in the ordinary light image. In addition, the image processing unit 204 may also include a feature quantity image generating unit, which acquires and displays a feature quantity image as a medical image (medical image), and generates the feature quantity image by calculation based on at least one of an ordinary light image obtained by irradiating light of a white band or irradiating light of a plurality of wavelength bands as the white band, and a special light image obtained by irradiating light of a specific wavelength band. In addition, the above-mentioned processing is performed under the control of the CPU 210.

[0078] <Implementation of various processor functions>

[0079] The functions of each part of the above-mentioned image processing unit 204 can be implemented using various processors (processors) and recording media. Among the various processors, there is a general-purpose processor, i.e., CPU (Central Processing Unit), which realizes various functions by executing software (programs). In addition, among the above-mentioned various processors, there is also a processor specifically used for image processing, i.e., GPU (Graphics Processing Unit), and a processor whose circuit structure can be changed after manufacturing FPGA (Field Programmable Gate Array), i.e., programmable logic device (PLD). In the case of learning or recognizing images as in the present invention, a structure using a GPU is effective. Moreover, processors, i.e., dedicated circuits, etc., having a circuit structure specially designed for executing specific processing such as ASIC (Application Specific Integrated Circuit) are also included in the above-mentioned various processors.

[0080] The functions of each part can be implemented by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of CPU and FPGA, or a combination of CPU and GPU). In addition, multiple functions can be implemented by one processor. As an example of implementing multiple functions with one processor, first, there is a form such as a computer, which uses a combination of one or more CPUs and software to form a processor, and the processor is implemented as multiple functions. Secondly, there is a form such as a system on chip (System On Chip: SoC), which uses a processor that implements the functions of the entire system with an IC (Integrated Circuit) chip. In this way, more than one of the above-mentioned various processors is used as a hardware structure to implement various functions. Moreover, more specifically, the hardware structure of these various processors is a circuit (circuitry) that combines circuit elements such as semiconductor elements. These circuits can also be circuits that use logical sum, logical product, logical negation, exclusive or, and logical operations that combine them to implement the above functions.

[0081] When the processor or circuit executes software (program), the computer-readable code of the executed software (for example, various processors or circuits constituting the image processing unit 204, and / or their combination) is pre-stored in a non-transitory recording medium such as ROM211 (ROM: Read Only Memory), and the computer refers to the software. The software pre-stored in the non-transitory recording medium includes a program for executing the medical image processing method (operation method of the medical image processing device) involved in the present invention and data used during execution (data related to the acquisition of medical images, specific data for notification conditions and notification methods, parameters used in the size estimation unit, etc.). Instead of storing the code in ROM211, it is also possible to store it in various non-transitory recording media such as optical magneto-recording devices and semiconductor memories. When the software is used for processing, for example, RAM212 (RAM: Random Access Memory) is used as a temporary storage area, and for example, data stored in EEPROM (Flectrically Erasable and Programmable Read Only Memory) not shown in the figure can also be referred to. The recording unit 207 may also be used as a “non-transitory recording medium”.

[0082] In addition, ROM 211 (ROM: Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code of a program that causes CPU 210 (computer) and / or image processing unit 204 (computer) to execute various image processing methods (including the medical image processing method involved in the present invention). RAM 212 (RAM: Random Access Memory) is a storage element for temporary storage during various processing, and can also be used as a buffer during image acquisition. The sound processing unit 209 outputs messages (sounds) related to medical image processing, size estimation, notification, etc. from the speaker 209A (notification unit, notification device, speaker) under the control of CPU 210 and image processing unit 204.

[0083] <Operation Section>

[0084] The operation unit 208 may be composed of a keyboard, a mouse, or other devices not shown in the figure. The user may, through the operation unit 208, instruct to execute medical image processing or specify conditions required for execution (e.g., the setting of notification conditions and notification methods described later). The operation through the operation unit 208 includes the setting of notification conditions and notification methods (see Figure 6 , 7), indicating the operation of completing the acquisition of the medical image of the part to be photographed. The above-mentioned operation recognition unit 232 recognizes the user operation via the operation unit 208, and performs processing in each unit of the CPU 210 and the image processing unit 204 according to the recognized operation.

[0085] <Information recorded in the recording section>

[0086] like Figure 4 As shown, an endoscopic image 260 (image, endoscopic image, medical image, medical image), a size estimation result 262 (size estimation result), an estimation state judgment result 264 (estimation state judgment result, accuracy), a reference image 266 (reference image; an image showing a shooting state suitable for size estimation), etc. are recorded in the recording unit 207 (recording device, storage device, non-temporary recording medium). The reference image 266 is an image showing a shooting state suitable for the estimation of the size of the region of interest (an image of the region of interest captured in a state where the size estimation can be performed accurately), and an image acquired in a past examination can be used. The reference image 266 is preferably recorded in a database state in association with the type of part or region of interest, observation light, and other shooting conditions.

[0087] <Each process of the medical image processing method>

[0088] Figure 5 This is a flowchart showing an outline of the processing of the medical image processing method (operation method of the medical image processing apparatus) according to the first embodiment.

[0089] <Setting of notification conditions and notification method>

[0090] The image processing unit 204 (notification control unit 228) sets notification conditions and notification methods according to user operations via the operation unit 208 (step S100: notification condition setting step, notification method setting step). Figure 6 The illustrated screen 700 (displayed on the monitor 400 ) performs setting operations.

[0091] Screen 700 has areas 702 to 712 configured with radio buttons and area 714 configured with a numerical input field. The user can set whether to notify (on or off; area 702) by operating the radio buttons. In addition, the user can set "whether to notify" (area 702), "whether to notify the accuracy of size estimation" (area 704), "whether to notify based on screen display" (area 706), "whether to display reference image" (area 708) by operating the radio buttons. Moreover, the user can set "whether to overlap and display operation auxiliary information on the endoscopic image (whether to display it in the first area or the second area)" (area 710), and "whether to notify based on sound output" (area 712) by operating the radio buttons. In addition, "operation auxiliary information" is information used to improve the result of the judgment of "whether the area of ​​interest in the endoscopic image is suitable for the estimation of size", and the result of the judgment can be improved by the user performing operations in accordance with the operation auxiliary information.

[0092] Furthermore, the user can set the "time from the start of notification to the end (from the notification state to the non-notification state)" by inputting a value in area 714. After the time (number of seconds) inputted in area 714 has passed, the notification control unit 228 switches the notification by the monitor 400 and / or the speaker 209A from the notification state to the non-notification state (stops or ends the notification). Figure 6 In the example of , the time from the start of the notification to the end is 2.0 seconds, but a different time may be set. The numerical input may also be a method of selecting a determined numerical value from a pull-down menu. By switching to the non-notification state, the assistance can be ended according to the needs of the user and excessive assistance can be suppressed. In addition, the notification control unit 228 may reduce (reduce) the notification force after a specified time has passed in addition to or instead of the end of the notification.

[0093] Apart from Figure 6 In addition to the screen 700, you can also Figure 7 The screen 750 shown sets the operation support information (first information to fifth information) to be notified. Figure 7In the screen 750 of FIG. 7 , the user can set “whether to notify the amount of movement of the endoscope 100” (area 752; first information), “whether to notify the direction of movement of the endoscope 100” (area 754; first information), “whether to notify the air supply to be turned on or off” (area 756; second information), “whether to notify the water supply to be turned on or off” (area 758; second information) by operating radio buttons. In addition, the user can set “whether to notify the lighting to be turned on or off” (area 760; third information), “whether to notify the image processing to be turned on or off” (area 762; fourth information), and “whether to notify whether the treatment tool can be used” (area 764; fifth information) by operating radio buttons. Regarding the air supply, water supply, lighting, and image processing, not only turning on or off but also the degree can be set through the screen.

[0094] The direction and / or amount of movement of the endoscope viewer 100 to be moved is the “first information”, the on / off of the air supply and / or the water supply (and its degree) is the “second information”, the on / off of the lighting and / or its degree is the “third information”, the on / off of the image processing for the acquired image and / or its degree is the “fourth information”, and whether the treatment instrument should be used is the “fifth information”.

[0095] By moving the endoscope observation device 100, the user can change the shooting direction (shooting angle) or distance to shoot the area of ​​interest from the front, or move the area of ​​interest closer to the reference position (for example, the center). By supplying air or water, it is possible to remove residues, pigments, etc., and improve the clarity of the area including the area of ​​interest. By lighting, the brightness or clarity of the image can be improved, and the observation light (ordinary light / special light, etc.) suitable for size estimation can be switched. In addition, by image processing, blur or jitter, for example, can be eliminated. By stopping the use of the treatment instrument at a necessary timing (making it retreat into the top hard part 116), the treatment instrument can be prevented from overlapping with the area of ​​interest and hindering the size estimation. By reporting such first to fifth information based on the user's settings, the area of ​​interest can be captured in the endoscopic image in a state suitable for size estimation.

[0096] In this way, in the endoscope system 10 (medical image processing device, endoscope system), the user can set the notification conditions and notification methods as needed, and the notification control unit 228 performs appropriate notification (assistance) according to the setting content. In addition, the above example is an example of setting, and other items (such as notification based on light or vibration) can also be set. In addition, the setting of the notification conditions and notification methods can be performed not only at the beginning of medical image processing, but also at any timing during the processing. Moreover, the setting of the notification conditions and notification methods can be performed automatically by the endoscope system 10 without the user's operation.

[0097] <Acquisition and Display of Endoscopic Images>

[0098] The image acquisition unit 220 acquires a time series of endoscopic images (images, medical images, and medical images) (step S110: image acquisition process). The image acquisition unit 220 may acquire an endoscopic image captured by the endoscope 100, or may acquire an endoscopic image 260 recorded in the recording unit 207. When the image acquisition unit 220 acquires an endoscopic image captured by the endoscope 100, the recording control unit 234 may record (save) the acquired image in the recording unit 207 as the endoscopic image 260.

[0099] The display control unit 236 displays the acquired endoscopic image on the monitor 400 (step S120 : image display step).

[0100] <Selection of Area of ​​Interest>

[0101] The region of interest selection unit 222 selects a region of interest to be the object of size estimation in the acquired endoscopic image (step S130: region of interest selection process). When the endoscopic image has two or more regions of interest, the region of interest selection unit 222 determines one object for size estimation based on the image. For example, the region of interest selection unit 222 may determine the region of interest with the largest area as the object, or may determine the region of interest with the highest accuracy of size estimation (described later in step S140) as the object. In addition, the region of interest selection unit 222 may determine the object based on the user operation via the operation unit 208.

[0102] In addition, the region of interest selection unit 222 may also detect the region of interest from the endoscopic image using a neural network (not shown) such as a CNN (Convolutional Neural Network). CNN has, for example, an input layer, an intermediate layer, and an output layer, and the intermediate layer calculates feature quantities by convolution operations or pooling processing. The output layer may include a fully connected layer, and outputs the position information of the object (region of interest) based on the feature quantities calculated by the intermediate layer. The region of interest selection unit 222 may also detect the region of interest by image processing other than a neural network.

[0103] <Judgment of Shooting Status>

[0104] The estimated state determination unit 226 determines the shooting state (accuracy of size estimation) of the selected region of interest by, for example, image processing (step S140: estimated state determination process). In the case where there are multiple regions of interest, the accuracy may be calculated for the multiple regions of interest, and the region of interest may be selected based on the result. When estimating the size of a region of interest such as a lesion, the shooting state of the region of interest has a great influence. For example, it is difficult to accurately estimate the size under the following shooting state. Therefore, the accuracy is calculated in consideration of these situations.

[0105] In addition, as described in detail later, in the endoscope system 10, by reporting operation auxiliary information used to improve the judgment result (accuracy), the shooting state or position and shape of the area of ​​interest can be maintained in a state suitable for size estimation, thereby shortening the endoscopic examination, reducing the burden, and ensuring the consistency of the estimation result.

[0106] (1) The area of ​​interest is not clearly captured

[0107] Endoscopes are difficult to operate, and fixed observation of lesions requires skills. Figure 8 The focus shift (blur) or motion blur (shaking) shown in parts (a) and (b) of FIG. 8 respectively deteriorates the image capturing state of the region of interest 802, and the accuracy of size estimation decreases. In addition, in the case of an endoscopic image, as shown in FIG. Figure 8 As shown in parts (c) and (d) of the image processing unit 226, the focus area 802 may be blocked by water supply or residue 804. Therefore, the clarity of the shooting state can be reflected in the accuracy. As "image processing" for calculating the above-mentioned accuracy, the estimated state judgment unit 226 can, for example, obtain the clarity of the edge based on the brightness signal and judge the focus offset, or judge the motion blur by comparing the size of the motion vector based on the frame close to the shooting timing. In addition, the estimated state judgment unit 226 can consider the presence or absence of the operation of the air and water supply button 141, and the difference in color or brightness between the focus area and the residue.

[0108] (2) When the area of ​​interest is not captured at the appropriate location

[0109] The wide viewing angle of the endoscope will cause aberration problems unique to the optical system. For example, Fig. 9 As shown in part (a) of FIG. 8 , when the region of interest 802 is near the center of the observed image 800, the distortion is small ( Fig. 9In the figure, the distortion is small in the low grayscale part and large in the higher grayscale part), but as shown in part (b) of the figure, when observing the peripheral part of the image 800, the distortion becomes large, which affects the size estimation of the region of interest 802. Therefore, the estimation state determination unit 226 can reflect the distance (number of pixels) between the reference position in the image (for example, the center of the image) and the region of interest in the accuracy.

[0110] (3) The shape of the region of interest in the image is inappropriate

[0111] Since the tissues in the body, such as the large intestine and stomach, are soft, the shape of the region of interest is always changing. In addition, in endoscopic examinations, the region of interest is mostly found and observed from the tangent direction. However, in this state, it is sometimes impossible to determine the exact size or shape of the region of interest. Therefore, the shooting angle (for example, the angle between the normal direction of the region of interest and the optical axis direction of the photographic optical system 130), the similarity with a pre-specified figure (circle, ellipse, etc.), etc. can be reflected in the accuracy. Fig.10 802 is an example of determining the accuracy based on the circularity of the region of interest 802. Part (a) of the figure shows an example of a circle 900 (a perfect circle) which is a reference for determining the circularity, indicated by a dotted line. Fig.10 As shown in the ellipse 902 in part (b), the ratio of the two axes can be any value (the figure used as the reference for determination can be a perfect circle or an ellipse). The ratio can also be changed according to the ease of change of the imaging part, etc.

[0112] The accuracy may be expressed as a continuous numerical value (eg, 0% to 100%) or a discrete numerical value (eg, an integer from 1 to 5, A to E, etc.) The recording control unit 234 may record the accuracy as the estimated state determination result 264 in the recording unit 207 .

[0113] The estimated state determination unit 226 determines whether the region of interest is suitable for estimation of size based on the determination result (accuracy) in step S140 (step S150: estimated state determination process). The estimated state determination unit 226 can determine whether the region of interest is suitable for estimation of size by comparing the calculated accuracy with a preset threshold value (for example, it can be set to 70%, but is not limited to this value). The estimated state determination unit 226 can also set the threshold value based on a value input by the user. If the determination result in step S150 is that the region of interest is suitable for estimation of size (YES in step S150), the process proceeds to step S180. If it is not suitable, the process proceeds to step S160.

[0114] <Estimation of the size of the area of ​​interest>

[0115] The size estimation unit 224 estimates the size of the area of ​​interest (step S180: size estimation process). The "size" in the present invention is not the size on the image (number of pixels), but the size in the actual space (2mm, 3mm, etc.). Therefore, general distance measurement technology (triangulation distance method or time-of-flight method) can be used: measurement of parallax using multi-viewpoint shooting with a stereo camera, etc., measurement of reflection or phase difference based on laser or ultrasonic waves, etc. In addition, in recent years, research has been conducted on accurately measuring distance based on a single camera device, so distance measurement using deep learning can also be used. In the measurement using deep learning, the input is a captured image, any one of the sensor signal information that can be obtained from the above-mentioned devices, or a combination thereof, and the output is the size in the actual space. The learning data is the size in the actual space with high accuracy based on the above-mentioned equipment or the visual observation of an expert.

[0116] In addition, the size estimation unit 224 may also perform estimation by comparison with an object of known size. As such an "object of known size", for example, a "scale of known size protruding from the forceps opening 126 by user operation" may be used. In addition, depending on the structure of the endoscope or the imaging optical system, a scale based on a laser beam as described in WO2018 / 159292 or Japanese Patent Application Publication No. 2019-195643 may be used.

[0117] The notification control unit 228 notifies the estimation result through the speaker 209A (speaker, notification device) and / or the monitor 400 (display, notification device) (step S190: notification step).

[0118] <Generation and Display of Operation Support Information>

[0119] If the judgment result (accuracy) in step S140 is not suitable for size estimation, the notification control unit 228 generates operation assistance information for improving the judgment result of the estimation state (step S160: generation process), and notifies the operation assistance information from the speaker 209A and / or the monitor 400 (step S170: notification process). The operation assistance information may be information indicating the operation of the endoscope 100 itself, or information indicating a state suitable for size estimation. In addition, the operation assistance information may include at least one of the first to fifth information described above.

[0120] <Method of notification (Part 1)>

[0121] Fig.11 The portion (a) is due to the fact that the region of interest 802 is hidden by the residue 804 ( Figure 8In the example of the state shown in part (d) of the figure, a message 904 of "Please supply water" is displayed on the monitor 400 (operation of the endoscope 100 itself; an example of the second information), and part (b) of the figure is an example of an arrow 906 (graphic) indicating the direction in which the endoscope 100 should be moved because the area of ​​interest is the end of the field of view (operation of the endoscope 100 itself; an example of the first information). These examples are examples of displaying (notifying) in the second area that does not overlap with the endoscopic image. The notification control unit 228 can display these operation assistance information on a screen different from the endoscopic image, or on a monitor different from the monitor used to display the endoscopic image. In addition, the notification control unit 228 can also notify the third to fifth information (operation assistance information related to the use of lighting, image processing, and treatment instruments) according to the setting in step S100.

[0122] exist Fig.11 In the example of FIG. 1 , the notification is performed in the area (second area) that does not overlap with the observed image, but the notification control unit 228 may also perform notification according to the setting content of the notification condition (refer to Figure 6 The area 710) is notified in the area (first area) within the observed image. Fig.12 Part (a) is an example of displaying an arrow 906 in an observation image, and part (b) of the figure is an example of overlapping a circle 900 (graphic) representing the shape of a region of interest suitable for size estimation on the observation image, and then outputting a voice of “Please shoot from the front” from the speaker 209A.

[0123] <Method of Notification (Part 2)>

[0124] Depending on the observation or inspection situation, it is sometimes unclear whether the user can fully understand the state of the estimated fit size only through the instructions of the operation as exemplified in the above-mentioned "notification method (one)". Therefore, more specifically, it is also conceivable to improve the understanding by searching for a reference image (an image representing the estimated shooting state of the fit size; a method of operation auxiliary information) close to the current image from the database (reference image 266 recorded in the recording unit 207) and simultaneously presenting the reference image. "A state close to the current image" refers to a state such as being close to the shooting part of the current image, close to the shooting conditions, close to the type of the area of ​​interest, close to the state of the image, and specifically refers to a state of being close to the shooting part, the brightness of the image, the color tone, the wavelength of the observation light, the size or shape of the area of ​​interest, the type of the endoscope, the shooting date and time, etc. Fig.13 This figure shows an example in which the notification control unit 228 displays an arrow 906 indicating the operation direction of the endoscope 100 and a reference image 266 (an image in which the region of interest is captured in a clear state near the center of the image) in the second region.

[0125] <Method of Notification (Part 3)>

[0126] The above-mentioned "notification methods (one) and (two)" are examples of notifying the operation assistance information separately, but by simultaneously notifying the accuracy of size estimation and shooting status information (for example, image clarity, the presence or absence of blur or jitter and its degree, the presence or absence of water supply or residue, the usage status of disposal instruments, etc.), it is possible to easily understand how the endoscope system 10 identifies the area of ​​interest and the adjustments made in accordance with the operation assistance information. Fig.14 Part (a) of the figure shows a case where the accuracy (in this case, 81%) is numerically displayed in addition to the arrow 906 and the circle 900 indicating the operation direction of the endoscope 100, and part (b) of the figure shows a case where the accuracy is displayed in a bar graph 908 in the second area (the length of the colored portion indicates the accuracy, with 0% at the left end and 100% at the right end). The notification control unit 228 can output the accuracy and imaging status information by voice.

[0127] <Method of Notification (Part 4)>

[0128] Fig.15 FIG. 2 is a diagram showing a case where the estimated result and accuracy of the size are reported in addition to the operation support information. Fig.15 In the example shown in part (a) of the figure, the estimated size (2 mm) is displayed in a dark color because the accuracy of the size estimation is low (23%), but in the example shown in part (b) of the figure, the estimated size (4 mm) is clearly displayed in white because the accuracy of the size estimation is high (74%). In this way, the notification control unit 228 can change the notification method when the accuracy is low and when the accuracy is high.

[0129] <Recognition of user operation and notification of judgment result>

[0130] Depending on the notification method of the operation assistance information, there is a possibility that the user may not be able to instantly understand the content, so the endoscope system 10 determines whether the user has performed the operation according to the operation assistance information and can also notify the user of the meaning. Fig.16 As shown in the flowchart of Figure 3 ) identifies the user's operation on the operation assistance information (step S172: operation identification process), determines whether the identified user operation complies with the operation assistance information (step S174: judgment process), and the notification control unit 228 notifies the judgment result through the monitor 400 and / or the speaker 209A (step S176: notification process).

[0131] Fig.179 is a diagram showing an example of notification of the judgment result of the user operation, and part (a) of the diagram shows the state before the user performs the operation (the state where the operation auxiliary information is notified). In this state, the accuracy of the size estimation result is low (23%), and the user needs to move the endoscope observation device 100 forward and backward, bend, etc. according to arrow 906 (one form of operation auxiliary information) to make the area of ​​interest near the center of the endoscope image. Fig.17 Part (b) is an example of the notification status after the user's operation. Compared with part (a) of the figure, the position of the focus area is closer to the center (reference position), so the operation recognition unit 232 judges that "the user operation complies with the operation assistance information", and the notification control unit 228 outputs a sound such as "Right! (Correct!)" from the speaker 209A (judgment result). Fig.17 The Cc) portion of the figure is also an example of the notification state after the user operation, but in this example, compared with the (a) portion of the figure, the position of the focus area is not close to the center (instead, the accuracy is reduced), so the operation recognition unit 232 determines that "the user operation does not comply with the operation auxiliary information", and the notification control unit 228 outputs a sound such as "Wrong! (error!)" from the speaker 209A (judgment result). Through such notification, the user can easily understand whether his or her operation complies with the operation auxiliary information, and can also assist the user in accurately estimating the size of the focus area.

[0132] In addition, Figure 5 In the flowchart, when the area of ​​interest is suitable for the estimated size (YES in step S150), the estimated result of the size is reported, but the notification control unit 228 can also report the estimated result in the same way when the area of ​​interest is not suitable for the estimated size (N0 in step S150).

[0133] The CPU 210 and the image processing unit 204 repeatedly perform the processing of steps S110 to S190 until the observation is completed (while the answer is NO in step S200 ).

[0134] As described above, according to the medical image processing apparatus, the endoscope system, the medical image processing method, and the program according to the present invention, it is possible to assist the user in accurately estimating the size of the region of interest.

[0135] (Note)

[0136] In addition to the above-described embodiments and examples, the following structures are also included in the scope of the present invention.

[0137] (Note 1)

[0138] A medical image processing device, wherein:

[0139] The medical image analysis processing unit detects a region of interest as a region of interest based on feature quantities of pixels of the medical image.

[0140] The medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit.

[0141] (Note 2)

[0142] A medical image processing device, wherein:

[0143] The medical image analysis processing unit detects whether there is an object of attention based on the feature quantity of the pixels of the medical image.

[0144] The medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit.

[0145] (Note 3)

[0146] A medical image processing device, wherein:

[0147] The medical image analysis result acquisition unit acquires the analysis result of the medical image from the recording device for recording.

[0148] The analysis result is either or both of the region of interest as a region of interest included in the medical image and the presence or absence of an object of interest.

[0149] (Note 4)

[0150] A medical image processing device, wherein:

[0151] The medical image is an ordinary light image obtained by irradiating light of a white band or irradiating light of a plurality of wavelength bands as light of a white band.

[0152] (Note 5)

[0153] A medical image processing device, wherein:

[0154] Medical images are images obtained by irradiating light with a specific wavelength band.

[0155] The specific wavelength band is a band narrower than the white wavelength band.

[0156] (Note 6)

[0157] A medical image processing device, wherein:

[0158] The specific wavelength band is the blue or green band of the visible range.

[0159] (Note 7)

[0160] A medical image processing device, wherein:

[0161] The specific wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and light in the specific wavelength band has a peak wavelength in the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

[0162] (Note 8)

[0163] A medical image processing device, wherein:

[0164] The specific wavelength band is the red band of the visible range.

[0165] (Note 9)

[0166] A medical image processing device, wherein:

[0167] The specific wavelength band includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and light in the specific wavelength band has a peak wavelength in the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

[0168] (Note 10)

[0169] A medical image processing device, wherein:

[0170] The specific wavelength band includes a wavelength band where the absorption coefficients of oxidized hemoglobin and reduced hemoglobin are different, and light of the specific wavelength band has a peak wavelength in the wavelength band where the absorption coefficients of oxidized hemoglobin and reduced hemoglobin are different.

[0171] (Note 11)

[0172] A medical image processing device, wherein:

[0173] The specific wavelength band includes 400±10nm, 440±10nm, 470±10nm, or a wavelength band above 600nm and below 750nm, and the light in the specific wavelength band has a peak wavelength in a wavelength band above 400±10nm, 440±10nm, 470±10nm, or a wavelength band above 600nm and below 750nm.

[0174] (Note 12)

[0175] A medical image processing device, wherein:

[0176] Medical images are images taken inside a living body.

[0177] The in-vivo image contains information on fluorescence emitted by fluorescent substances in the body.

[0178] (Note 13)

[0179] A medical image processing device, wherein:

[0180] Fluorescence is obtained by irradiating a living body with excitation light having a peak value of 390 nm to 470 nm.

[0181] (Note 14)

[0182] A medical image processing device, wherein:

[0183] Medical images are images taken inside a living body.

[0184] The specific wavelength band is a wavelength band of infrared light.

[0185] (Note 15)

[0186] A medical image processing device, wherein:

[0187] The specific wavelength band includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and light in the specific wavelength band has a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

[0188] (Note 16)

[0189] A medical image processing device, wherein:

[0190] The medical image acquisition unit includes a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on an ordinary light image obtained by irradiating light of a white band or irradiating light of a plurality of wavelength bands as light of a white band.

[0191] Medical images are special light images.

[0192] (Note 17)

[0193] A medical image processing device, wherein:

[0194] The signal of a specific wavelength band is obtained by calculation based on the color information of RGB or CMY included in the normal light image.

[0195] (Note 18)

[0196] A medical image processing device, wherein:

[0197] A feature quantity image generating unit is provided for generating a feature quantity image by calculating at least one of an ordinary light image obtained by irradiating light of a white band or irradiating light of a plurality of wavelength bands as the white band, and a special light image obtained by irradiating light of a specific wavelength band,

[0198] Medical images are feature quantity images.

[0199] (Note 19)

[0200] An endoscope device, comprising:

[0201] The medical image processing device as described in any one of Notes 1 to 18; and

[0202] An endoscope acquires an image by irradiating at least either light of a white wavelength band or light of a specific wavelength band.

[0203] (Note 20)

[0204] A diagnostic aid device, wherein:

[0205] A medical image processing device as described in any one of Notes 1 to 18.

[0206] (Note 21)

[0207] A medical service auxiliary device, wherein:

[0208] A medical image processing device as described in any one of Notes 1 to 18.

[0209] Although the embodiments and other examples of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0210] Explanation of symbols

[0211] 10 Endoscopic system

[0212] 100 Endoscope Viewer

[0213] 102 Hand side operation

[0214] 104 Insertion

[0215] 106 General Cable

[0216] 108 Optical connector

[0217] 112 Soft Part

[0218] 114 Bend

[0219] 116 Top hard part

[0220] 116A Top side end face

[0221] 123 Lighting Department

[0222] 123A Lighting lens

[0223] 123B Lighting Lens

[0224] 126 Clamp mouth

[0225] 130 Photographic optical system

[0226] 132 Photographic lens

[0227] 134 Camera Components

[0228] 136 drive circuit

[0229] 138 AFE

[0230] 139 Observer information recording unit

[0231] 141 Gas and water supply buttons

[0232] 142 Attraction Button

[0233] 143 Function buttons

[0234] 144 Shooting button

[0235] 170 Light Guide

[0236] 200 processors

[0237] 202 Image Input Controller

[0238] 204 Image Processing Department

[0239] 205 Communication Control Department

[0240] 206 Video output unit

[0241] 207 Records Department

[0242] 208 Operation Department

[0243] 209 Sound Processing Department

[0244] 209A Speaker

[0245] 210 CPU

[0246] 211 ROM

[0247] 212 RAM

[0248] 220 Image acquisition unit

[0249] 222 Focus Area Selection Department

[0250] 224 Dimension Estimation Department

[0251] 226 Estimated state determination unit

[0252] 228 Notification Control Department

[0253] 230 Observer information acquisition unit

[0254] 232 Operation recognition unit

[0255] 234 Recording Control Unit

[0256] 236 Display control unit

[0257] 260 Endoscopic images

[0258] 262 Size estimation results

[0259] 264 Estimated status judgment result

[0260] 266 reference images

[0261] 300 Light source device

[0262] 310 Light Source

[0263] 310B blue light source

[0264] 310G Green Light Source

[0265] 310R Red light source

[0266] 310V purple light source

[0267] 330 aperture

[0268] 340 Condenser lens

[0269] 350 Light source control unit

[0270] 400 Monitor

[0271] 700 screens

[0272] 702 Area

[0273] 704 Area

[0274] 706 Area

[0275] 708 Area

[0276] 710 Area

[0277] 712 Area

[0278] 714 Area

[0279] 750 frames

[0280] 752 Area

[0281] 754 Area

[0282] 756 Area

[0283] 758 Area

[0284] 760 Area

[0285] 762 Area

[0286] 764 Area

[0287] 800 Observation images

[0288] 802 Area of ​​concern

[0289] 804 Residue

[0290] 900 yen

[0291] 902 Ellipse

[0292] 906 Arrow

[0293] 908 Graphics

[0294] S100~S200 are the steps of the medical image processing method.

Claims

1. A medical image processing device, comprising a processor, wherein: The processor acquires a time series of images, calculating the accuracy of the estimation of the size of the region of interest by image processing according to the clarity of the region including the region of interest in the image, By comparing the accuracy with a preset threshold, it is determined whether the region of interest is suitable for the estimated size. generating operation assistance information for improving the result of the judgment based on the result of the judgment, The result of the determination and the operation assistance information are notified by a notifying device.

2. A medical image processing device, comprising a processor, wherein: The processor acquires a time series of images, calculating the accuracy of the estimation of the size of the region of interest by image processing based on the distance between the reference position in the acquired image and the region of interest in the image, By comparing the accuracy with a preset threshold, it is determined whether the target area is suitable for the estimated size, and based on the result of the determination, operation assistance information for improving the result of the determination is generated, The result of the determination and the operation assistance information are notified by a notifying device.

3. A medical image processing device, comprising a processor, wherein: The processor acquires a time series of images, calculating the accuracy of the estimation of the size of the region of interest by image processing according to the shooting angle when the region of interest in the image is shot, By comparing the accuracy with a preset threshold, it is determined whether the region of interest is suitable for the estimated size. generating operation assistance information for improving the result of the judgment based on the result of the judgment, The result of the determination and the operation assistance information are notified by a notifying device.

4. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The processor notifies the accuracy through the notifying device.

5. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The medical image processing apparatus comprises: a storage device storing a reference image representing an estimated imaging state suitable for the size; The processor causes at least one of the notification devices to display the reference image.

6. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The processor causes the notification device to display the result of the determination and the operation assistance information in a first area within the acquired image and / or a second area that does not overlap with the acquired image.

7. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The processor determines a target for estimating the size based on an image having two or more of the regions of interest.

8. The medical image processing apparatus according to claim 7, wherein: The processor determines the region of interest with the largest area among the two or more regions of interest as the object.

9. The medical image processing apparatus according to claim 7, wherein: The processor determines, as the target, a region of interest having the highest estimation accuracy of the size among the two or more regions of interest.

10. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The processor estimates the size of the region of interest and notifies the estimation result through the notification device.

11. The medical image processing apparatus according to any one of claims 1 to 3, wherein: The processor identifies the user's operation, determines whether the identified user's operation complies with the operation assistance information, and notifies the result of the determination through the notification device.

12. An endoscope system comprising: The medical image processing device according to any one of claims 1 to 11; The notification device; and an endoscope which is inserted into a subject as an object to be examined and captures the image, The processor acquires the image captured by the endoscope.

13. The endoscope system according to claim 12, wherein: The notification device includes a display for displaying information on a screen and / or a speaker for outputting sound. The processor uses at least one of graphics, text, and sound to notify the result of the determination and the operation assistance information through the display and / or the speaker.

14. The endoscope system according to claim 12 or 13, wherein: The operation assistance information includes at least one of the following information: first information indicating the direction and / or the amount of movement of the endoscope to be moved; second information indicating on / off of air supply and / or water supply from the endoscope; third information indicating on / off of illumination from the endoscope and / or the degree of illumination; fourth information indicating on / off of image processing for the acquired image and / or the degree of image processing; and fifth information indicating whether a treatment instrument should be used.

15. The endoscope system according to claim 12 or 13, wherein: The processor acquires individual information of the endoscope scope, The determination and / or the notification are performed based on the individual information.

16. A medical image processing method, comprising: An image acquisition process for acquiring time series images; an accuracy calculation step of calculating the accuracy of the estimation of the size of the region of interest by image processing based on the clarity of the region including the region of interest in the image; An estimation state judgment step, judging whether the region of interest is suitable for size estimation by comparing the accuracy with a preset threshold; an operation assistance information generating step of generating operation assistance information for improving the result of the judgment based on the result of the judgment; as well as In the notification step, a notification device is used to notify the result of the judgment and the operation assistance information.

17. A medical image processing method, comprising: An image acquisition process for acquiring time series images; an accuracy calculation step of calculating the accuracy of the estimation of the size of the region of interest by image processing based on the distance between the reference position in the acquired image and the region of interest in the image; An estimation state judgment step, judging whether the region of interest is suitable for size estimation by comparing the accuracy with a preset threshold; an operation assistance information generating step of generating operation assistance information for improving the result of the judgment based on the result of the judgment; as well as In the notification step, a notification device is used to notify the result of the judgment and the operation assistance information.

18. A medical image processing method, comprising: An image acquisition process for acquiring time series images; an accuracy calculation step of calculating the accuracy of the estimation of the size of the region of interest by image processing according to the shooting angle when the region of interest in the image is shot; An estimation state judgment step, judging whether the region of interest is suitable for size estimation by comparing the accuracy with a preset threshold; an operation assistance information generating step of generating operation assistance information for improving the result of the judgment based on the result of the judgment; as well as In the notification step, a notification device is used to notify the result of the judgment and the operation assistance information.

19. A recording medium which is non-transitory and computer-readable, The recording medium stores a program for causing a computer to execute the medical image processing method according to any one of claims 16 to 18.

Images