Medical image processing device, operation method for medical image processing device, medical image processing program, and recording medium

Abstract

To provide a medical image processing device, an operation method for a medical image processing device, a medical image processing program, and a recording medium that are capable of accurately estimating a state of an observation area of an observation image captured by an endoscope.SOLUTION: In a medical image processing device with a processor 22, an image acquisition unit 110 of the processor 22 acquires an observation image 100 of an observation area in a human body captured by an endoscope. A distance information acquisition unit 112 of the processor 22 acquires (estimates) distance information for a distance between the endoscope and the observation area from the observation image 100. A state estimation unit 114 of the processor 22 estimates a state of the observation area based on the observation image 100 and the distance information.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a medical image processing apparatus, a method for operating a medical image processing apparatus, a medical image processing program, and a recording medium, and particularly relates to a technique for accurately estimating the state of an observation region from an observation image obtained by endoscopy of an observation region inside the body.

Background Art

[0002] Conventionally, in order to prevent overlooking lesions during endoscopy, lesions are automatically detected and notified by AI (Artificial Intelligence) or the like. There is also a discriminative AI or the like for determining whether a detected lesion is a true lesion.

[0003] Patent Document 1 proposes an endoscope processor with high ability to detect lesions.

[0004] The endoscope processor described in Patent Document 1 generates a first processed image and a second processed image by performing different image processes on a captured image captured by an endoscope, and uses a learning model that inputs the generated first processed image and second processed image to output the state of a disease, which is the estimation result of the learning model.

[0005] Patent Document 1 also describes that a learning model is generated for each disease to be diagnosed or a learning model is generated for each part to be diagnosed. The user selects and instructs which disease or which part to be diagnosed, and the endoscope processor outputs the state of the disease using the learning model corresponding to the user's selection instruction.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] Incidentally, when estimating the condition of an internally observed area (for example, the presence or absence of a lesion) from observational images taken using an endoscope, the accuracy of the estimation may decrease depending on the distance at which the image was taken. This is thought to be because the appearance of the observational image differs depending on the distance at which it was taken.

[0008] The present invention has been made in view of these circumstances, and aims to provide a medical image processing device, a method for operating the medical image processing device, a medical image processing program, and a recording medium that can accurately estimate the state of the observation area of ​​an observation image taken by an endoscope. [Means for solving the problem]

[0009] To achieve the above objective, the invention according to the first embodiment is a medical image processing apparatus equipped with a processor, wherein the processor acquires observation images of an observation area inside the body taken by an endoscope, acquires distance information relating to the distance from the endoscope to the observation area, and estimates the state of the observation area based on the observation images and the distance information.

[0010] According to a first aspect of the present invention, when estimating the state of an observation image taken of an observation area inside the body using an endoscope, distance information relating to the distance from the endoscope to the observation area can also be used to accurately estimate the state of the observation area regardless of the distance at which the observation image was taken.

[0011] In a medical image processing apparatus according to a second aspect of the present invention, it is preferable that the processor selects one of a plurality of estimation modes for estimating the state of the observation area based on acquired distance information, and estimates the state of the observation area using the selected estimation mode.

[0012] In a medical image processing apparatus according to a third aspect of the present invention, it is preferable that the processor obtains a final output by weighting and averaging the estimation results of the state of the observation area, each estimated by a plurality of estimation modes for estimating the state of the observation area, according to the acquired distance information.

[0013] This allows the estimation results of the state of the observed area, estimated by multiple estimation modes, to be weighted and averaged using an optimal weighting coefficient, and the state of the observed area can be accurately estimated from this weighted average.

[0014] In a medical image processing apparatus according to a fourth aspect of the present invention, the plurality of estimation modes preferably include a near-view mode that estimates the state of an observation area based on an observation image, and a far-view mode that estimates the state of an observation area based on an observation image, wherein the far-view mode has higher accuracy in estimating the state of the observation area than the near-view mode when the distance information exceeds a threshold. Since the appearance of an observation image differs depending on the distance, it is preferable to apply an estimation mode (near-view mode, far-view mode) that is suitable for the distance of the observation image.

[0015] In a medical image processing apparatus according to a fifth aspect of the present invention, the apparatus includes a first learning model that takes an observation image as input and estimates the state of an observation area corresponding to the near-view mode, and a second learning model that takes an observation image as input and estimates the state of an observation area corresponding to the far-view mode, wherein the processor preferably estimates the state of the observation area using at least one of the first learning model and the second learning model.

[0016] In a medical image processing apparatus according to a sixth aspect of the present invention, it is preferable that the plurality of estimation modes include a detection mode for detecting lesions present in an observation area based on an observation image, and a differentiation mode for classifying lesions present in an observation area into two or more classes based on an observation image.

[0017] In a medical image processing apparatus according to a seventh aspect of the present invention, the observed image includes a normal light image captured using normal light and a special light image captured using special light. Preferably, the processor acquires an observation mode indicating whether the observed image is a normal light image or a special light image, along with distance information of the observed image, and selects a detection mode or a discrimination mode based on the distance information and the observation mode. Since normal light images and special light images appear differently, it is preferable to also use information from these images to select the detection mode or discrimination mode.

[0018] In the medical image processing apparatus according to the eighth aspect of the present invention, it is preferable that the processor selects a discrimination mode when the distance information is below a threshold and the observation mode is a special light observation mode for observing a special light image.

[0019] In a medical image processing apparatus according to the ninth aspect of the present invention, it is preferable that the processor acquires distance information corresponding to a plurality of sub-regions in the observation area, and estimates the state of each of the sub-regions in the observation area based on the observation image and the distance information corresponding to the plurality of sub-regions. A sub-region may be a single pixel in the observation image, or a region consisting of multiple pixels.

[0020] In a medical image processing apparatus according to a tenth aspect of the present invention, it is preferable that the processor inputs the acquired observation image to the third learning model and obtains the distance information estimated by the third learning model. Since distance information is obtained from the observation image, physical distance measurement using laser light or the like for distance measurement is unnecessary, and distance information of the observation area in the observation image can be obtained even in endoscopes and the like where it is difficult to mount new measuring instruments.

[0021] In the medical image processing apparatus according to the eleventh aspect of the present invention, there is a fourth learning model that inputs at least one of a distance map indicating distance information of an observation region of an observation image and the observation image, and outputs a weighting coefficient used for calculation of weighted average. It is preferable that the processor inputs at least one of the distance map and the observation image to the fourth learning model, and acquires the weighting coefficient used for calculation of weighted average from the fourth learning model.

[0022] In the medical image processing apparatus according to the twelfth aspect of the present invention, it is preferable that the processor causes a display device that displays an observation image to display the estimated state of the observation region.

[0023] In the medical image processing apparatus according to the thirteenth aspect of the present invention, it is preferable that the processor detects a lesion existing in the observation region, classifies the lesions existing in the observation region into two or more classes, recognizes a treatment tool existing in the observation region, or recognizes an organ or a part existing in the observation region, as the state of the observation region.

[0024] The invention according to the fourteenth aspect is an operation method of a medical image processing apparatus including a processor, the method including: a step in which the processor acquires an observation image obtained by photographing an observation region inside the body with an endoscope; a step in which the processor acquires distance information regarding the distance from the endoscope to the observation region; and a step in which the processor estimates the state of the observation region based on the observation image and the distance information.

[0025] In the operation method of the medical image processing apparatus according to the fifteenth aspect of the present invention, the step of estimating the state of the observation region preferably includes: a step of selecting any one of a plurality of estimation modes for estimating the state of the observation region based on the acquired distance information; and a step of estimating the state of the observation region according to the selected estimation mode.

[0026] In the method of operating a medical image processing apparatus according to the 16th aspect of the present invention, in the step of acquiring distance information, it is preferable to acquire distance information corresponding to a plurality of small regions in the observation region respectively, and in the step of estimating the state of the observation region, the state of each of the plurality of small regions in the observation region is estimated respectively based on the observation image and the distance information corresponding to the plurality of small regions.

[0027] The invention according to the 17th aspect is a medical image processing program for causing a computer to execute the method of operating a medical image processing apparatus according to any one of the 14th to 16th aspects.

[0028] The invention according to the 18th aspect is a non-temporary and computer-readable recording medium on which the medical image processing program according to the 17th aspect is recorded.

Effect of the Invention

[0029] According to the present invention, from an observation image obtained by photographing an observation region inside the body with an endoscope, the state of the observation region can be accurately estimated regardless of the photographing distance of the observation image.

Brief Description of the Drawings

[0030] [Figure 1] FIG. 1 is a system configuration diagram of an endoscope system including a processor device that functions as a medical image processing apparatus according to the present invention. [Figure 2] FIG. 2 is a block diagram showing an embodiment of the hardware configuration of the processor device constituting the endoscope system shown in FIG. 1. [Figure 3] FIG. 3 is a functional block diagram showing a first embodiment of the main processor of the processor device shown in FIG. 2. [Figure 4] FIG. 4 is a diagram showing an example of an observation image and a distance map obtained from the observation image. [Figure 5] FIG. 5 is a diagram showing an example of observation images of a long shot, a medium shot, and a close shot. [Figure 6] FIG. 6 is a block diagram showing a specific example of the processor shown in FIG. 3. [Figure 7] Figure 7 shows an example of a distant view observation image to which the distant view mode is applied, and a close-up observation image to which the close-up mode is applied. [Figure 8] Figure 8 shows an example of an observation image to which mode switching is applied, and a distance map showing distance information of the observation area of ​​that observation image. [Figure 9] Figure 9 shows the relationship between the switching patterns between near-view and far-view modes and the estimation accuracy. [Figure 10] Figure 10 shows an example of a user interface when the user sets the switching pattern between close-up mode and far-away mode. [Figure 11] Figure 11 shows an example of weighting coefficients (near view ratio, far view ratio) used for weighted averaging of the estimated lesion area results estimated in near view mode and the estimated lesion area results estimated in far view mode. [Figure 12] Figure 12 is a block diagram showing another specific example of the processor shown in Figure 3. [Figure 13] Figure 13 is a functional block diagram showing a second embodiment of the main processor of the processor device shown in Figure 2. [Figure 14] Figure 14 is a flowchart showing an embodiment of the operating method of the medical image processing device according to the present invention. [Modes for carrying out the invention]

[0031] Hereinafter, preferred embodiments of the medical image processing apparatus, the operating method of the medical image processing apparatus, the medical image processing program, and the recording medium according to the present invention will be described with reference to the attached drawings.

[0032] [System Configuration] Figure 1 is a system configuration diagram of an endoscope system including a processor device that functions as a medical image processing device according to the present invention.

[0033] In Figure 1, the endoscope system 1 consists of an endoscope 10, a processor device 20, a light source device 30, and a display device 40.

[0034] Endoscope 10 is also called an electronic endoscope or endoscopic scope. Endoscope 10 also includes laparoscopes.

[0035] The endoscope 10 captures images of the observation area inside the subject's body and acquires endoscopic images (hereinafter referred to as "observation images"), which are medical images. The tip of the endoscope 10 contains an optical system (objective lens) and an image sensor, and light from the observation area enters the image sensor through the objective lens. The image sensor converts the light from the observation area that enters its imaging surface into an electrical signal and outputs an image signal that represents the observation image.

[0036] The rear end of the endoscope 10 is provided with a video connector and a light guide connector that connect the endoscope 10 to the processor unit 20 and the light source unit 30. By attaching the video connector on the endoscope 10 to the processor unit 20, an image signal indicating the observation image captured by the endoscope 10 is sent to the processor unit 20. Furthermore, by attaching the light guide connector on the endoscope 10 to the light source unit 30, illumination light emitted from the light source unit 30 is irradiated from the illumination window on the tip surface of the endoscope 10 toward the observation area via the light guide connector and a light guide installed inside the endoscope 10.

[0037] The light source device 30 supplies illumination light to the light guide of the endoscope 10, which is fitted with a light guide connector, via the light guide connector. Depending on the observation mode selected by the user (for example, normal light observation mode, special light observation mode, etc.), the light source device 30 can emit normal light (white broadband light or multiple broadband lights with different wavelength bands) or one or more special lights (light with a specific narrowband, or a combination thereof, or light with various wavelength bands depending on the observation purpose).

[0038] Furthermore, when normal light is emitted from the light source device 30, the endoscope 10 can capture a normal light image (WL (White Light) image), and when special light is emitted from the light source device 30, the endoscope 10 can capture a special light image (BLI (Blue Light Imaging or Blue LASER Imaging) image) or an LCI (Linked Color Imaging) image).

[0039] The special light used for BLI is an observation light with a high proportion of V (Violet) light, which has a high absorption rate in superficial blood vessels, and a low proportion of G (Green) light, which has a high absorption rate in intermediate blood vessels. This makes it suitable for generating images (BLI images) that are well-suited for highlighting blood vessels and structures in the superficial layer of the mucous membrane of the subject.

[0040] Furthermore, the special light used for LCI has a higher ratio of V light compared to the observation light used for WL, making it more suitable for capturing subtle color changes. LCI images are color-enhanced images that utilize the R (Red) component signal to enhance the colors, especially around mucous membranes, making reddish colors appear redder and whitish colors appear whiter.

[0041] [Processor device] Figure 2 is a block diagram showing an embodiment of the hardware configuration of the processor device that constitutes the endoscope system shown in Figure 1.

[0042] The processor device 20 shown in Figure 2 consists of an image acquisition unit 21, a processor 22, a memory 23, a display control unit 24, an input / output interface 25, and an operation unit 26.

[0043] The image acquisition unit 21 includes a connector to which the video connector of the endoscope 10 is connected, and acquires observation images captured by the image sensor located at the tip of the endoscope 10 via the connector. The processor unit 20 also acquires remote signals from the operation of the handheld control unit of the endoscope 10 via the connector to which the endoscope 10 is connected. The remote signals include a release signal that instructs still image capture, an observation mode signal that indicates the observation mode, and so on.

[0044] The observation images can be videos or still images taken in sync with the shutter release signal.

[0045] The processor 22 consists of a CPU (Central Processing Unit) and other components, and controls all parts of the processor device 20. It also functions as a processing unit that performs image processing of observation images acquired from the endoscope 10, AI (Artificial Intelligence) processing to estimate the state of the observation area from the observation images, processing to acquire distance information regarding the distance from the endoscope 10 to the observation area, and acquiring and saving still images using the release signal acquired via the endoscope 10.

[0046] Memory 23 includes flash memory, ROM (Read-only Memory), RAM (Random Access Memory), and a hard disk drive. The flash memory, ROM, or hard disk drive is a non-volatile memory that stores various programs executed by the processor 22. The RAM functions as a workspace for processing by the processor 22 and also temporarily stores programs stored in the flash memory, etc. The processor 22 may also have a portion of memory 23 (RAM) built into it. Still images taken during endoscopic examinations can also be saved in memory 23.

[0047] The display control unit 24 generates a display image based on the processed observation image (moving image or still image) from the processor 22 and the AI-processed state estimation result of the observation area, and outputs the display image to the display device 40.

[0048] The input / output interface 25 includes a connection unit for wired and / or wireless connection to external devices, and a communication unit that can connect to a network. By transmitting observed images to an external device such as a personal computer connected via the input / output interface 25, the external device may be made to have some or all of the functions of the medical image processing apparatus according to the present invention.

[0049] Furthermore, a foot switch (not shown) is connected to the input / output interface 25. The foot switch is an operating device placed at the operator's feet and operated by foot, and by pressing the pedal, it sends an operation signal to the processor unit 20. The processor unit 20 is connected to storage (not shown) via the input / output interface 25. The storage (not shown) is an external storage device connected to the processor unit 20 via a LAN (Local Area Network), etc., and is, for example, a file server for a medical image filing system such as PACS (Picture Archiving and Communication System), or a NAS (Network Attached Storage).

[0050] The control unit 26 includes a power switch, switches for manually adjusting white balance, light intensity, zoom, etc., and switches for setting observation modes.

[0051] [First Embodiment of a Processor] Figure 3 is a functional block diagram showing a first embodiment of the main processor of the processor device shown in Figure 2.

[0052] As shown in Figure 3, the processor 22 includes an image acquisition unit 110, a distance information acquisition unit 112, and an observation area state estimation unit 114.

[0053] The image acquisition unit 110 is the part that acquires the observation images 100 acquired from the endoscope 10 by the image acquisition unit 21 of the processor device 20. The image acquisition unit 21 of the processor device 20 is the hardware part that acquires the observation images 100 from the endoscope 10, while the image acquisition unit 110 of the processor 22 is the software part that acquires the observation images 100 for processing by the subsequent distance information acquisition unit 112 and state estimation unit 114. Therefore, the image acquisition unit 110 also includes cases where the observation images 100 have been temporarily stored in the memory 23 and then acquired from the memory 23.

[0054] The distance information acquisition unit 112 is the part that receives the observation image 100 acquired by the image acquisition unit 110 and acquires distance information regarding the distance from the endoscope 10 (tip) to the observation area based on the input observation image 100.

[0055] In this example, the distance information acquisition unit 112 uses a learning model (third learning model) to estimate distance information, inputs the observed image 100 into the third learning model, and acquires the distance information estimated by the third learning model.

[0056] Figure 4 shows an example of an observed image and a distance map obtained from that image.

[0057] Figure 4A shows the observation image input to the fourth learning model used by the distance information acquisition unit 112, and Figure 4B shows an example of a distance map showing the distance estimation results estimated by the fourth learning model.

[0058] The distance map shown in Figure 4B assigns pixel values ​​such that the further the distance from the endoscope 10 to the observation area, the darker the pixel value, and the closer the distance, the whiter the pixel value.

[0059] The third learning model used by the distance information acquisition unit 112 is a Generative Adversarial Network (GAN) having a generator and a discriminator, to which a pre-trained generator, learned unsupervised from a large number of observed images, can be applied. The fourth learning model is not limited to a generator learned by a GAN; for example, it may be one learned by a Variational Autoencoder (VAE).

[0060] The observation area state estimation unit 114 receives the observation image 100 acquired by the image acquisition unit 110 and the distance information acquired (estimated) from the observation image 100 by the distance information acquisition unit 112. Based on the observation image 100 and the distance information, the observation area state estimation unit 114 estimates the state of the observation area.

[0061] Figure 5 shows examples of observation images of distant, mid-range, and close-up views.

[0062] The appearance of the mucous membrane in the observation area during endoscopy can vary greatly depending on the distance, even for the same area.

[0063] Figure 5 illustrates how a lesion appears when observed from a distance, mid-range, and close up. The characteristics of the lesion at each distance are as follows:

[0064] Figure 5A is a distant view image. From this distant view image, an oval-shaped lesion with a strong reddish tint (the area of ​​high density in Figure 5A) can be observed compared to the surrounding area.

[0065] Figure 5B is a close-up observation image, capturing the circular area surrounding the lesion included in the distant observation image (Figure 5A). From this close-up observation image, a strong redness (higher density area in Figure 5B) and irregular mucosal pattern can be observed compared to the surrounding area.

[0066] Figure 5C is a close-up observation image, capturing the circular area surrounding the lesion included in the mid-range observation image of Figure 5B. From this close-up observation image, irregularities in the mucosal pattern can be observed throughout the entire image.

[0067] Thus, even the same lesion can exhibit different characteristics depending on the distance from which it is imaged. Therefore, it may be inappropriate to estimate the state of the observed area, including lesion detection and differentiation, using the same AI at all distances.

[0068] Therefore, the observation area state estimation unit 114 selects one of several estimation modes for estimating the state of the observation area based on the distance information acquired from the distance information acquisition unit 112, estimates the state of the observation area using the selected estimation mode, or calculates the final output by weighting the estimation results of the state of the observation area estimated by each of the multiple estimation modes according to the acquired distance information.

[0069] Here, "estimation of the state of the observation area" includes the presence or absence of lesions in the observation area and / or detection (estimation) of lesion areas, classification of lesions in the observation area into two or more classes (e.g., neoplastic, non-neoplastic), recognition of instruments in the observation area (presence or absence of instruments and / or recognition of the type of instruments), or recognition of organs or parts in the observation area. If the endoscope 10 is an upper gastrointestinal endoscope, recognition of organs in the observation area means detecting (classifying) organs such as the esophagus, stomach, and duodenum, and recognition of parts means, for example, if the organ is the stomach, detecting various parts of the stomach such as the cardia, fundus, upper body, middle body, lower body, antrum, pyloric antrum, and pylorus.

[0070] In the following example, we will explain how to "estimate the state of the observed area," specifically how to detect (estimate) lesions present in the observed area.

[0071] <Specific example of the processor of the first embodiment> Figure 6 is a block diagram showing a specific example of the processor shown in Figure 3.

[0072] As shown in Figure 6, the observed image 100 is input to the processor 22. The processor 22 reads out the learned model 23A and the necessary parameter groups from the input observed image 100 and the learned model 23A, first parameter group 23B, second parameter group 23C, third parameter group 23D, fourth parameter group 23E, ... stored in memory 23, and functions as the distance information acquisition unit 112 and the observation area state estimation unit 114 shown in Figure 3.

[0073] One or more known learning models can be applied as the learning model 23A, and the learning model used as the state estimation unit 114 can be, for example, a convolutional neural network (CNN).

[0074] Furthermore, the first parameter group 23B, the second parameter group 23C, the third parameter group 23D, the fourth parameter group 23E, etc., are applied to one or more learning models 23A, and include, for example, the filter coefficients of the filters used in the convolutional layers of the CNN, and the weight coefficients for the inputs of each layer of the CNN.

[0075] The processor 22 uses the third set of parameters applied to the learning model 23A read from memory 23 as a trained third learning model, and by inputting the observed image 100 into this third learning model, it estimates and outputs distance information indicating the distance of the observed region.

[0076] Furthermore, this example includes two estimation modes for estimating the lesion area: a near-view mode and a far-view mode. Here, the far-view mode is a mode in which the accuracy of lesion area estimation is higher than that of the near-view mode when the distance information exceeds a certain threshold.

[0077] When the processor 22 performs lesion area estimation in near-view mode, it applies a first set of parameters to the learning model 23A read from memory 23, and uses it as a learning model (first learning model) to estimate lesion areas in near-view observation images 100 that have distance information below a threshold. By inputting the observation images 100 into this first learning model, the processor estimates lesion areas in near-view observation images 100.

[0078] On the other hand, when the processor 22 performs lesion area estimation using the distant view mode, it applies a second set of parameters to the same learning model 23A read from memory 23, and uses it as a learning model (second learning model) to estimate lesion areas in the distant view observation image 100 that has distance information exceeding a threshold. By inputting the observation image 100 into this second learning model, the processor estimates lesion areas in the distant view observation image 100.

[0079] The first set of parameters can be obtained by training an untrained model 23A with a dataset of numerous training data sets, each consisting of pairs of close-up observation images (see 5C in Figure 4) and ground truth data (ground truth mask) indicating the lesion area. Similarly, the second set of parameters can be obtained by training an untrained model 23A with a dataset of numerous training data sets, each consisting of pairs of distant observation images (see 5A in Figure 4) and ground truth data indicating the lesion area.

[0080] Figure 7 shows an example of a distant view observation image to which the distant view mode is applied, and a close-up observation image to which the close-up mode is applied.

[0081] If the processor 22 determines that the input observation image 100 is a foreground observation image 100 with distance information below a threshold, based on the output of a third learning model that estimates distance information (especially when the distance is uniform throughout the image, as shown in 7B of Figure 7), it selects the foreground mode as the estimation mode for estimating the lesion area, and uses the first learning model corresponding to the selected foreground mode to estimate the lesion area within the foreground observation image 100.

[0082] Furthermore, if the processor 22 determines that the input observation image 100 is a distant observation image 100 with distance information exceeding a threshold, based on the output of the third learning model that estimates distance information (especially when the distance is uniform throughout the image, as shown in 7A of Figure 7), it selects the distant view mode as the estimation mode for estimating the lesion area, and uses the second learning model corresponding to the selected distant view mode to estimate the lesion area within the distant observation image 100.

[0083] Then, if the near-view mode is selected, the processor 22 uses the first learning model corresponding to the near-view mode to estimate the lesion area in the near-view observation image 100 and outputs it as an estimation result. If the far-view mode is selected, the processor 22 uses the second learning model corresponding to the far-view mode (in this example, by switching the parameter group used for learning model 23A from the first parameter group 23B to the second parameter group 23C) to estimate the lesion area in the far-view observation image 100 and outputs it as an estimation result.

[0084] Furthermore, the processor 22 may not only switch the estimation mode (the learning model used) depending on whether the observation image is a close-up or a distant view, as described above, to output the estimated results of the lesion area, but may also weight the estimated results of the lesion area estimated by multiple estimation modes (in this example, close-up mode and distant view mode) according to the distance information of the observation image 100 and use that as the final output.

[0085] Figure 8 shows an example of an observation image to which mode switching is applied, and a distance map showing distance information of the observation area of ​​that observation image.

[0086] The observation image shown in Figure 8, 8A, includes the distant observation area in the upper left and the close-up observation area in the lower right. The distance map shown in Figure 8, 8A, assigns pixel values ​​such that the further the distance from the endoscope to the observation area, the darker the pixel value; the closer the distance, the whiter the pixel value.

[0087] In the case of such observational images, it is preferable to switch between applying the estimation mode within the observational image between near-view mode and far-view mode, rather than applying either near-view mode or far-view mode to the entire image.

[0088] In this example, the processor 22 applies the distant view mode to the upper left region of the observation image shown in 8A of Figure 8, and the close-up view mode to the other regions, and then integrates the estimated results of the lesion region estimated using both estimation modes and outputs them.

[0089] Furthermore, the area of ​​the observation image to which the near-view mode or far-view mode is applied may include one or more enclaves depending on the distance information of the observation area.

[0090] Figure 9 shows the relationship between the switching patterns between near-view and far-view modes and the estimation accuracy.

[0091] Figure 9C is a distance map showing the distance from the endoscope to the observation area estimated from the observation image.

[0092] Figure 9A shows the case where a certain threshold (first threshold) is set for the distance information shown in the distance map in Figure 9C, and the near-view mode is applied to the region of the entire observed image that has distance information below the first threshold, and the far-view mode is applied to the region that has distance information above the first threshold, in order to estimate the lesion area.

[0093] According to the switching pattern between foreground and background modes shown in Figure 9A, the area to which background mode is applied is larger than the area to which foreground mode is applied.

[0094] On the other hand, Figure 9B shows the case where a threshold smaller than the first threshold (second threshold) is set for the distance information shown in the distance map in Figure 9C, and the near-view mode is applied to the areas of the entire observed image that have distance information less than the second threshold, and the far-view mode is applied to the areas that have distance information greater than the second threshold, in order to estimate the lesion area.

[0095] According to the switching pattern between foreground and background modes shown in Figure 9B, the area to which background mode is applied is smaller than the area to which foreground mode is applied.

[0096] Furthermore, the estimation results obtained by applying the switching pattern shown in Figure 9A to estimate the lesion area were more accurate than the estimation results obtained by applying the switching pattern shown in Figure 9B to estimate the lesion area.

[0097] Therefore, in the case of the observation image in this example, it is preferable to apply the switching pattern shown in 9A of Figure 9 (i.e., to set the area to which the foreground mode is applied and the area to which the background mode is applied using the first threshold).

[0098] To estimate lesion areas with greater accuracy, it is necessary to set an optimal switching pattern (i.e., the optimal threshold for distinguishing between areas to which near-view and far-view modes should be applied). Possible methods for determining the optimal threshold include either determining it in advance through trial and error, or automatically estimating the optimal threshold for each observed image using a learning model and then using that estimated threshold.

[0099] In this case, the learning model can be generated by machine learning an untrained learning model using a dataset of numerous training data sets, each pairing an observed image with a threshold (ground truth data) for which a highly accurate estimation of the lesion area was obtained for that observed image.

[0100] Figure 10 shows an example of a user interface when the user sets the switching pattern between close-up mode and far-away mode.

[0101] Figure 10A shows an example of a distance map that displays distance information for the observation area of ​​an observed image.

[0102] As shown in Figures 10B and 10C, the processor 22 displays a distance map on the display device 40 and, adjacent to the distance map, displays an operation screen having a slider (triangular icon) for setting thresholds.

[0103] The user can set threshold A shown in Figure 10, 10B, and threshold B shown in Figure 10, 10C, by moving the slider to a position of any desired density (distance) using the control unit 26 or a touch panel (not shown).

[0104] Furthermore, as shown in Figures 10B and 10C, when a threshold A or threshold B is set by the user, the processor 22 displays on the distance map, in a way that distinguishes between the areas to which the foreground mode is applied and the areas to which the background mode is applied, by coloring or the like, according to threshold A or threshold B.

[0105] This allows users to check the switching patterns between near-view and far-view modes based on thresholds and set their desired switching pattern.

[0106] Furthermore, in the above example, the areas to which the near-view mode is applied and the areas to which the far-view mode is applied are completely separated from the observed image, and the estimated results of the lesion area are obtained using a learning model corresponding to each mode. However, the method is not limited to this, and the estimated results of the lesion area estimated in near-view mode and the estimated results of the lesion area estimated in far-view mode may be weighted and averaged according to the distance information to obtain the final output.

[0107] Figure 11 shows an example of weighting coefficients (near view ratio, far view ratio) used for weighted averaging of the estimated lesion area results estimated in near view mode and the estimated lesion area results estimated in far view mode.

[0108] Figure 11, section 11C, shows an example of a distance map that displays distance information for the observation area of ​​an observed image. The processor 22 calculates the near view ratio and far view ratio on a pixel-by-pixel basis based on the distance map. Note that the near view ratio and far view ratio may be calculated not only on a pixel-by-pixel basis, but also on a unit of a smaller area larger than one pixel.

[0109] As shown in Figure 11, 11A, the near view factor and far view factor in this example are in the range of 0.0 to 1.0, and the sum of the near view factor and far view factor obtained for a given pixel is 1.0.

[0110] The processor 22 uses a learning model (fourth learning model) that takes a distance map as input and outputs weighting coefficients (near-view factor, far-view factor) used in weighted averaging calculations. The distance map can be input to the fourth learning model, and the near-view factor and far-view factor for each pixel of the observed image can be obtained from the fourth learning model.

[0111] Figure 11, section 11B is an illustrative diagram showing the near-field and far-field percentages for each pixel of an observed image acquired based on the distance map in Figure 11, section 11C.

[0112] In Figure 6, the processor 22 applies the fourth parameter group to the learning model 23A read from memory 23 to use it as a trained fourth learning model, inputs the distance map into the fourth learning model, and obtains the near view and far view ratios for each pixel of the observed image estimated by the fourth learning model.

[0113] Furthermore, the fourth learning model may be generated by taking an observed image as input instead of a distance map and estimating and outputting the near-field and far-field ratios for each pixel of the observed image, or by taking both an observed image and a distance map as input and estimating and outputting the near-field and far-field ratios for each pixel of the observed image.

[0114] Alternatively, the processor 22 may obtain the near-field ratio and far-field ratio based on the distance information for each pixel from a lookup table in which the near-field ratio and far-field ratio are stored in advance according to the distance information, without using the fourth learning model.

[0115] The processor 22 obtains the estimated results of the lesion area estimated in near view mode, the estimated results of the lesion area estimated in far view mode, and the near view ratio and far view ratio for each pixel of the observed image, and then calculates a weighted average based on the following formula.

[0116] [Mathematics 1] Weighted average = (Estimated result using distant view mode) × (Distant view rate) + (Estimated result using close-up mode) × (Close-up ratio) The processor 22 displays the estimated lesion area (state of the observed area) obtained from the observed image 100 as described above on the display device 40 that displays the observed image 100. For example, if a lesion area is detected from the observed image 100, the estimated lesion area can be displayed on the display device 40 by superimposing a bounding box or the like surrounding the lesion area onto the observed image 100.

[0117] <Other specific examples of the processor of the first embodiment> Figure 12 is a block diagram showing another specific example of the processor shown in Figure 3.

[0118] The processor 22 shown in Figure 6 reads the learning model 23A from memory 23 and applies the first parameter group 23B, second parameter group 23C, third parameter group 23D, fourth parameter group 23E, ... to the learning model 23A to acquire distance information; the weighting coefficient acquisition unit acquires weighting coefficients used for weighted averaging of the estimated results of lesion areas estimated in near view mode and the estimated results of lesion areas estimated in far view mode; and switches between the first parameter group 23B and the second parameter group 23C to operate the state estimation unit in near view mode or far view mode. However, the processor 22 shown in Figure 12 differs from the processor 22 shown in Figure 6 in that it individually includes a distance information acquisition unit 120, a weighting coefficient acquisition unit 122, a first state estimation unit 124, a second state estimation unit 126, ...

[0119] Here, the first state estimation unit 124 and the second state estimation unit 126 use learning models (first learning model, second learning model) to which the first parameter group 23B and the second parameter group 23C shown in Figure 6 are applied, respectively, and input the observed image 100 to the first learning model and the second learning model, respectively, to obtain the estimated results of the lesion area from the first learning model and the second learning model. Here, the first learning model can output estimated results of the lesion area with high detection accuracy for the near-view observation area of ​​the input observed image 100, and the second learning model can output estimated results of the lesion area with high detection accuracy for the far-view observation area of ​​the input observed image 100.

[0120] The processor 22 selects and outputs the estimated results of the lesion regions estimated by the first state estimation unit 124 and the second state estimation unit 126, respectively, based on the distance information acquired by the distance information acquisition unit 120. This output includes selecting one of the estimated results of the lesion regions for the entire observation image 100, integrating the estimated results of the selected lesion regions on a pixel-by-pixel or sub-region basis, or weighting and averaging the estimated results of the lesion regions estimated by the first state estimation unit 124 and the second state estimation unit 126 according to the distance information of the observation image 100 to obtain the final output.

[0121] [Second Embodiment of the Processor] Figure 13 is a functional block diagram showing a second embodiment of the main processor of the processor device shown in Figure 2.

[0122] As shown in Figure 13, the processor 22 includes a distance information acquisition unit 130, an observation mode acquisition unit 132, a detection mode / identification mode selection unit 134, a first state estimation unit 136, and a second state estimation unit 138.

[0123] The distance information acquisition unit 130 receives the observation image 100 as input and acquires distance information regarding the distance from the endoscope to the observation area based on the input observation image 100. The acquired distance information is then output to the detection mode / differentiation mode selection unit 134.

[0124] The observation mode acquisition unit 132 is responsible for acquiring the observation mode, which indicates whether the observation image 100 is a normal light image or a special light image. It outputs an observation mode signal indicating the acquired current observation mode to the detection mode / identification mode selection unit 134.

[0125] Furthermore, the observation mode acquisition unit 132 can acquire the current observation mode by receiving an observation mode signal, which is, for example, a remote signal from a user operation on the handheld control unit of the endoscope 10, indicating the observation mode.

[0126] The detection mode / identification mode selection unit 134 selects either the detection mode or the identification mode based on the input distance information and observation mode.

[0127] Here, the detection mode is one of several estimation modes, which detects lesions (lesion areas) present in the observation area based on the observation image, and the differential mode is another of several estimation modes, which classifies lesions present in the observation area into two or more classes (e.g., neoplastic, non-neoplastic) based on the observation image. The differential mode may include the detection mode, or it may be a mode that classifies the lesion areas detected by the detection mode individually.

[0128] The detection mode / identification mode selection unit 134 selects the identification mode if the input distance information is below a threshold and the observation mode is the special light observation mode for observing special light images; otherwise, it selects the detection mode.

[0129] When users observe using special light images taken at close range where the distance information is below a threshold, they may prefer to differentiate lesions rather than detect them. Accordingly, in cases such as "close-range imaging and special light observation mode," it is preferable to select the differentiation mode. Furthermore, observation images taken under the conditions of "close-range imaging and special light observation mode" clearly show the blood vessels and structures on the mucosal surface of the subject, making them suitable for differentiation.

[0130] The first state estimation unit 136 is a state estimation unit that operates when a detection mode is selected, and detects lesions present in the observation area based on the input observation image 100. The detection of lesions (lesion areas) by the first state estimation unit 136 can be performed using a learning model for detecting lesions, as described above. In addition, the first state estimation unit 136 may selectively acquire the estimation results of lesions estimated in near-view mode and far-view mode using distance information, or perform a weighted average using distance information as the final output.

[0131] The second state estimation unit 138 is a state estimation unit that operates when the differentiation mode is selected. Based on the input observation image 100, it classifies the lesions present in the observation area into two or more classes and outputs the classified classes (differentiation results).

[0132] Furthermore, the second state estimation unit 138 uses a pre-trained model that has been machine-trained using a dataset of numerous training data consisting of pairs of special light images taken at close range and the corresponding discrimination results (ground truth data) for those special light images. By inputting the observed image (special light image taken at close range) into this trained model, it can output the discrimination result. Alternatively, the second state estimation unit 138 may use a support vector machine (SVM), a type of machine learning model, to classify lesions.

[0133] The processor 22 selects either the first state estimation unit 136 or the second state estimation unit 138 according to the detection mode or differentiation mode selected by the detection mode / differentiation mode selection unit 134, and displays the lesion detection result or differentiation result estimated by the selected first state estimation unit 136 or second state estimation unit 138 on the display device 40 on which the observation image 100 is displayed. This allows the user to confirm the lesion detection result or differentiation result on the screen of the display device 40.

[0134] [How to operate a medical image processing device] Figure 14 is a flowchart showing an embodiment of the operating method of the medical image processing device according to the present invention.

[0135] The method of operating the medical image processing device is, for example, the method of operating a medical image processing device equipped with the processor 22 shown in Figure 3, and the processor 22 performs the following various processes according to the flowchart shown in Figure 14.

[0136] In Figure 14, the processor 22 acquires observational images of the internal observation area captured by the endoscope via the image acquisition unit 110 (step S10).

[0137] Next, the processor 22 acquires distance information regarding the distance from the endoscope to the observation area using the distance information acquisition unit 112 (step S12). In this example, the distance information acquisition unit 112 uses a learning model (third learning model) that estimates distance from the observation image, and acquires distance information from the third learning model by inputting the observation image into the third learning model.

[0138] The processor 22 estimates the state of the observation area (for example, the lesion area present in the observation area) based on the observation image acquired in step S10 and the distance information acquired in step S12 (step S14).

[0139] The state estimation unit 114 of the observation area of ​​the processor 22 selects one of several estimation modes (e.g., near view mode, far view mode) for estimating the lesion area based on distance information, estimates the lesion area using the selected estimation mode, or weights the estimation results of the lesion area estimated by each of the multiple estimation modes according to the acquired distance information and uses that as the final output.

[0140] Furthermore, in step S12, when distance information corresponding to multiple sub-regions within the observation area, including the pixel units of the observed image, is acquired, in step S14, which estimates the state of the observation area (lesion area), the state of each of the multiple sub-regions within the observation area is estimated based on the observed image and the distance information corresponding to the multiple sub-regions.

[0141] The processor 22 outputs the estimation result estimated in step S14 (step S16). Preferably, the processor 22 displays the estimation result on the display device 40 that displays the observed image.

[0142] Next, the processor 22 determines whether the user has finished observing the observed image (step S18). If the processor 22 determines that the observation has not finished, it proceeds to step S10 and repeatedly executes the processes from step S10 to step S18. That is, if the observed image is a video, the estimation results for the video observed image can be obtained in real time by repeatedly executing the processes from step S10 to step S18 every frame or every few frames. The determination of the end of observation can be made, for example, by detecting a user action that ends the endoscopic examination.

[0143] When the processor 22 determines in step S18 that the observation has ended, it terminates this process.

[0144] [others] In this embodiment, distance information regarding the distance from the endoscope to the observation area is obtained from the observation image using AI. However, this is not limited to this, and for example, if the endoscope is equipped with a distance measuring unit that physically measures the distance between the tip of the endoscope and the object to be observed using laser light or the like, the distance information measured by that distance measuring unit may be obtained.

[0145] Furthermore, in this embodiment, two estimation modes, a near-view mode and a far-view mode, were described as multiple estimation modes applied according to distance. However, the embodiment is not limited to these two modes. For example, three or more estimation modes, including a mid-range near-view mode applied to the mid-range near-view between the near-view and far-view, may be used to estimate the state of the observation area for observation images at corresponding distances.

[0146] Furthermore, the hardware structure that performs various controls for the medical image processing apparatus according to the present invention consists of various processors as shown below. These various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various control units; a Programmable Logic Device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing; and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which has a circuit configuration specifically designed to perform a particular process.

[0147] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, multiple control units may be composed of a single processor. Examples of composing multiple control units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as multiple control units, as is typical of computers such as clients and servers. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple control units, on a single IC (Integrated Circuit) chip, as is typical of a System on a Chip (SoC). Thus, various control units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0148] Furthermore, the present invention includes a medical image processing program that, when installed on a computer, causes the computer to function as a medical image processing device according to the present invention, and a non-temporary, computer-readable recording medium on which this medical image processing program is recorded.

[0149] Furthermore, it goes without saying that the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0150] 1 Endoscopy System 10 Endoscopes 20 Processor Units 21 Image acquisition unit 22 processors 23 memory 23A Learning Model 23B First Parameter Group 23C Second parameter group 23D Third Parameter Group 23E Fourth parameter group 24 Display Control Unit 25 Input / Output Interfaces 26 Control section 30 Light source device 40 Display device 100 observation images 110 Image acquisition unit 112 Distance information acquisition section 114 State Estimation Unit 120, 130 Distance information acquisition section 122 Weighting coefficient acquisition unit 124, 136 First state estimation unit 126, 138 Second state estimation unit 132 Observation mode acquisition unit 134 Detection Mode / Identification Mode Selection Section S10 Step S12 Step S14 Step S16 Step S18 Step

Claims

1. In a medical image processing device having a processor, The processor: Obtaining an observation image of an observation area inside the body using an endoscope; acquiring distance information relating to the distance from the endoscope to the observation region; estimating a state of the observation area based on the observation image and the distance information; Medical imaging equipment.

2. the processor selects one of a plurality of estimation modes for estimating the state of the observation area based on the acquired distance information, and estimates the state of the observation area using the selected estimation mode. The medical image processing device according to claim 1 .

3. the processor performs a weighted average of the estimation results of the state of the observation area, which are estimated using a plurality of estimation modes for estimating the state of the observation area, according to the acquired distance information, to obtain a final output. The medical image processing device according to claim 1 .

4. the plurality of estimation modes include a foreground mode in which a state of the observation area is estimated based on the observation image, and a background mode in which a state of the observation area is estimated based on the observation image, and when the distance information exceeds a threshold, the background mode estimates the state of the observation area with higher accuracy than the foreground mode. The medical image processing device according to claim 2 .

5. a first learning model that receives the observation image and estimates a state of the observation area corresponding to the close view mode; and a second learning model that receives the observation image and estimates a state of the observation area corresponding to the distant view mode, the processor estimates a state of the observation area using at least one of the first learning model and the second learning model; The medical image processing device according to claim 4 .

6. the plurality of estimation modes include a detection mode for detecting a lesion present in the observation area based on the observation image, and a discrimination mode for classifying the lesion present in the observation area into two or more classes based on the observation image.

4. The medical image processing device according to claim 2 or 3.

7. the observed image includes a normal light image captured using normal light and a special light image captured using special light, The processor: acquiring an observation mode indicating whether the observation image is the normal light image or the special light image together with the distance information of the observation image; selecting the detection mode or the discrimination mode based on the distance information and the observation mode; The medical image processing device according to claim 6 .

8. the processor selects the discrimination mode when the distance information is equal to or less than a threshold and the observation mode is a special light observation mode in which the special light image is observed. The medical image processing device according to claim 7 .

9. The processor: acquiring the distance information corresponding to each of a plurality of small regions in the observation region; estimating states of the plurality of small regions in the observation region based on the observation image and the distance information corresponding to the plurality of small regions; The medical image processing device according to claim 1 .

10. a third learning model that receives the observed image and estimates the distance information; the processor inputs the acquired observed image into the third learning model and acquires the distance information estimated by the third learning model. The medical image processing device according to claim 1 .

11. a fourth learning model that receives input of at least one of a distance map indicating the distance information of the observation region of the observation image and the observation image, and outputs a weighting coefficient used in calculating the weighted average; the processor inputs at least one of the distance map and the observed image into the fourth learning model, and obtains the weighting coefficients used in calculating the weighted average from the fourth learning model. The medical image processing device according to claim 3 .

12. the processor causes the estimated state of the observation area to be displayed on a display device that displays the observation image. The medical image processing device according to claim 1 .

13. the processor detects a lesion present in the observation area as the state of the observation area, classifies the lesion present in the observation area into two or more classes, recognizes a treatment tool present in the observation area, or recognizes an organ or a part present in the observation area. The medical image processing device according to claim 1 .

14. 1. A method of operating a medical imaging device having a processor, comprising: a step of the processor acquiring an observation image of an observation region inside a body by an endoscope; the processor obtaining distance information relating to a distance from the endoscope to the observation region; the processor estimating a state of the observation area based on the observation image and the distance information; A method of operating a medical imaging device, comprising:

15. The step of estimating a state of the observation area includes: selecting one of a plurality of estimation modes for estimating a state of the observation area based on the acquired distance information; and estimating a state of the observation area using the selected estimation mode.

15. A method for operating a medical imaging system according to claim 14.

16. The step of acquiring distance information includes acquiring the distance information corresponding to each of a plurality of small regions in the observation region, the step of estimating the state of the observation area includes estimating states of the plurality of small areas in the observation area based on the observation image and the distance information corresponding to the plurality of small areas; A method for operating a medical image processing apparatus according to claim 14 or 15.

17. A medical image processing program that causes a computer to execute the method for operating a medical image processing apparatus according to any one of claims 14 to 16.

18. A non-transitory computer-readable recording medium on which the medical image processing program according to claim 17 is recorded.

Images