Image processing device, image processing method and program

Abstract

This image processing device 1X is provided with a permeation distance acquisition means 32X and an output control means 33X. The permeation distance acquisition means 32X acquires a permeation distance of a tumor site in a subject in an endoscopic image obtained by the imaging of the subject by an imaging unit arranged in an endoscope, on the basis of the endoscopic image. The output control means 33X outputs an image or a sound in accordance with the permeation distance into an output device.

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that process images acquired during an endoscopic examination. [Background technology]

[0002]

[0003] Conventionally, an endoscopic examination system that displays an image of the inside of an organ lumen has been known. For example, Patent Document 1 discloses a method for supporting diagnosis of a disease using an endoscopic image of a digestive organ. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-078539 Summary of the Invention [Problem to be solved by the invention]

[0004] CAD (Computer Aided Detection / Diagnosis) technology has been proposed to assist in the detection and diagnosis of lesions from images taken during endoscopic examinations. However, when a tumor is present, the distribution of the tumor's infiltration level must be determined through detailed examinations such as biopsies.

[0005] In view of the above-mentioned problems, one object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of presenting information about a tumor during an endoscopic examination. [Means for solving the problem]

[0006] One aspect of the image processing device is Based on an endoscopic image of a subject captured by an imaging unit provided in the endoscope, At a given cutting surface The invasion distance of the tumor site of the subject and an estimation of the depth of each wall layer of the subject at the cut surface. Means and Estimated The infiltration distance and the estimated depth of each of the wall layers. Based on a cross-sectional view of the subject taken along the cutting plane, showing the tumor site and each of the wall layers; to the output device display output control means for controlling the The image processing device has:

[0007] One aspect of the image processing method includes: Based on an endoscopic image of a subject captured by an imaging unit provided in the endoscope, At a given cutting surface The invasion distance of the tumor site of the subject and the depth of each wall layer of the subject at the cross section. , Estimated The infiltration distance and the estimated depth of each of the wall layers. Based on a cross-sectional view of the subject taken along the cutting plane, showing the tumor site and each of the wall layers; to the output device display do, It is an image processing method.

[0008] One aspect of the program is Based on an endoscopic image of a subject captured by an imaging unit provided in the endoscope, At a given cutting surface The invasion distance of the tumor site of the subject and the depth of each wall layer of the subject at the cross section. , Estimated The infiltration distance and the estimated depth of each of the wall layers. Based on a cross-sectional view of the subject taken along the cutting plane, showing the tumor site and each of the wall layers; to the output device display It is a program that causes a computer to execute the process. [Effects of the Invention]

[0009] As an example of one effect of the present disclosure, information about tumors can be presented during endoscopic examination. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows a schematic configuration of an endoscopic examination system. [Figure 2] 1 shows the hardware configuration of an image processing device. [Figure 3] 10(A) to 10(D) are diagrams schematically illustrating the flow of display processing based on the infiltration distance of a tumor site. [Figure 4]FIG. 10 is a functional block diagram of an image processing device relating to display processing based on the invasion distance of a tumor site. [Figure 5] 1A shows an outline of the infiltration distance estimation process based on Example 1. FIG. 1B shows an outline of the infiltration distance estimation method along a cross-section specified line. [Figure 6] 1 shows a first display example of a display screen displayed by a display device during an endoscopic examination. [Figure 7] 10 shows a second display example of a display screen displayed by the display device during an endoscopic examination. [Figure 8] 1 is an example of a flowchart showing an outline of processing executed by an image processing device during endoscopic examination in the first embodiment. [Figure 9] FIG. 10 is a schematic configuration diagram of an endoscopic examination system according to a modified example. [Figure 10] FIG. 10 is a block diagram of an image processing device according to a second embodiment. [Figure 11] 10 is an example of a flowchart executed by the image processing device in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. As shown in Fig. 1, the endoscopic examination system 100 is a system that presents information about a site of a subject suspected of having a tumor (also referred to as a "tumor site") to an examiner such as a doctor who performs an examination or treatment using an endoscope, and mainly comprises an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1.

[0013] The image processing device 1 acquires images (also referred to as "endoscopic images Ia") captured by the endoscope 3 in time series from the endoscope 3, and displays a screen based on the endoscopic images Ia on the display device 2. The endoscopic images Ia are images captured at a predetermined frame rate during at least one of the steps of inserting or ejecting the endoscope 3 into the subject. In this embodiment, when the image processing device 1 detects an endoscopic image Ia containing a tumor site (also referred to as a "tumor-containing image"), it estimates the infiltration distance (i.e., tumor depth) of the tumor in the subject's region within the tumor-containing image, and displays an image based on the estimated infiltration distance on the display device 2. As will be described later, examples of the "image based on the infiltration distance" include a map showing the infiltration distance, a cross-sectional view at a cut plane specified by the user, and a three-dimensional model representing the three-dimensional shape of the tumor using CG (Computer Graphics).

[0014] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0015] The endoscope 3 mainly comprises an operation unit 36 ​​through which the examiner makes predetermined inputs, a flexible shaft 37 that is inserted into the subject's organ to be imaged, a tip 38 that incorporates an imaging unit such as a micro-imaging element, and a connection unit 39 for connecting to the image processing device 1. In this embodiment, the operation unit 36 ​​includes a button (also referred to as a "still image save button") that instructs the examiner to capture (i.e., save as a still image) the endoscopic image displayed on the display device 2 when the examiner determines that an endoscopic image including a tumor site has been displayed on the display device 2.

[0016] 1 is an example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may be configured from multiple devices.

[0017] Note that the subject of endoscopic examination in the present disclosure is not limited to the large intestine, but may be any organ that can be examined endoscopically, such as the esophagus, stomach, pancreas, etc. For example, examples of endoscopes that are applicable in the present disclosure include pharyngoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, angioscopes, and epidural endoscopes.

[0018] (2) Hardware Configuration 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0019] The processor 11 executes predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0020] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs that allow the image processing device 1 to execute each process in this embodiment.

[0021] The memory 12 also stores tumor detection model information D1 relating to a tumor detection model that detects an endoscopic image Ia that is a tumor-containing image from an input endoscopic image Ia, and infiltration distance estimation model information D2 relating to an infiltration distance estimation model that estimates the infiltration distance of a tumor site contained in the input image. The tumor detection model information D1 and the infiltration distance estimation model information D2 will be described later.

[0022] The interface 13 acts as an interface between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0023] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel stacked on the display device 2, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also have a built-in pump for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0024] Next, the tumor detection model information D1 and the infiltration distance estimation model information D2 stored in the memory 12 will be described in detail.

[0025] The tumor detection model information D1 is information about a tumor detection model that, when an endoscopic image Ia is input, outputs information about whether or not the input endoscopic image Ia contains a tumor site. The tumor detection model information D1 includes parameters necessary for configuring the tumor detection model. For example, the tumor detection model is a classification model that, when an endoscopic image Ia is input, outputs a classification result regarding the presence or absence of a tumor site in the input endoscopic image Ia. The tumor detection model may be any machine learning model (including statistical models, the same applies hereinafter) such as a neural network or a support vector machine. Representative models of such neural networks include, for example, a fully convolutional network, SegNet, U-Net, V-Net, a feature pyramid network, Mask R-CNN, and DeepLab. When the tumor detection model is configured using a neural network, the tumor detection model information D1 includes various parameters, such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weight of each element of each filter.

[0026] The infiltration distance estimation model information D2 is information about an infiltration distance estimation model that, when an image of a subject's body part including a tumor is input, estimates the infiltration distance of the tumor site in the image, and includes parameters necessary for configuring the infiltration distance estimation model. The infiltration distance estimation model is a model that learns the relationship between the image input to the infiltration distance estimation model and the infiltration distance of the tumor site of the subject depicted in the image. The infiltration distance estimation model may be, for example, any machine learning model (including statistical models; the same applies hereinafter) such as a neural network or a support vector machine. For example, when the infiltration distance estimation model is configured using a neural network, the infiltration distance estimation model information D2 includes various parameters such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weight of each element of each filter.

[0027] As will be described later, the image input to the infiltration distance estimation model may be a partial image of the tumor-containing image that has been regularly cut out (e.g., in a grid pattern) from the tumor-containing image, or it may be the tumor-containing image itself. For example, in the former case, the infiltration distance estimation model outputs a numerical value indicating the estimated infiltration distance at the center position of the input partial image, and in the latter case, the infiltration distance estimation model outputs an image indicating the estimated infiltration distance for each pixel (or for each block of multiple pixels or for each subpixel) of the entire input tumor-containing image.

[0028] Furthermore, the invasion distance estimation model may be a model that outputs, in addition to the invasion distance, an estimation result regarding the depth of each layer constituting the wall layer of the object shown in the image input to the invasion distance estimation model. For example, if the object is the large intestine, the invasion distance estimation model estimates the depth of each of the mucosal layer, muscularis mucosa, submucosa, muscularis proper, subserosa, and serosa layers, and if the object is the esophagus, the invasion distance estimation model estimates the depth of each of the mucosal layer, submucosa, muscularis proper, and adventitia layers. Note that the model that estimates the depth of each layer constituting the wall layer may be a model separate from the invasion distance estimation model.

[0029] Furthermore, when the tumor detection model and the infiltration distance estimation model are learning models, the tumor detection model and the infiltration distance estimation model are trained in advance based on a pair of an input image conforming to the input format of each model and correct answer data indicating the correct answer to be output when the input image is input to each model. Then, the parameters of each model obtained by training are stored in memory 12 as tumor detection model information D1 and infiltration distance estimation model information D2, respectively.

[0030] (3) Display processing based on infiltration distance The display process based on the invasion distance of the tumor site will be described.

[0031] (3-1) Overview In summary, when the image processing device 1 detects an endoscopic image Ia that is a tumor-containing image, it estimates the infiltration distance at each position on the subject depicted in the tumor-containing image and displays an image based on the estimated infiltration distance on the display device 2. This allows the image processing device 1 to present information regarding the infiltration distance of the tumor site to the examiner without requiring a detailed examination such as a biopsy. Therefore, the image processing device 1 can immediately present information required for determining whether surgery is necessary, etc., to the examiner during the endoscopic examination.

[0032] 3(A) to 3(D) are diagrams that schematically show the flow of display processing based on the infiltration distance of a tumor site.

[0033] First, as shown in Fig. 3(A), the image processing device 1 acquires time-series endoscopic images Ia from the endoscope 3. Then, as shown in Fig. 3(B), the image processing device 1 recognizes tumor-containing images among the acquired endoscopic images Ia by automatically detecting the tumor site using a tumor detection model or by a user specifying it using the still image save button on the operation unit 36, and displays the recognized tumor-containing images on the display device 2.

[0034] Next, the image processing device 1 displays the tumor-containing image on the display device 2, and receives input of a cross-section designation line "Lc" that designates a cross section of a location including the tumor site via the input unit 14, as shown in Fig. 3(C). The cross-section designation line Lc may be designated by, for example, a mouse input or a touch panel input.

[0035] Then, as shown in FIG. 3(D), the image processing device 1 displays on the display device 2 an image based on the infiltration distance for each position of the tumor-containing image estimated using the infiltration distance estimation model. In this case, the image processing device 1 displays at least one of the following images: a map of the infiltration distance of the subject corresponding to the endoscopic image Ia (also referred to as an "infiltration distance map"), a cross-sectional view of the subject taken along the cross-section designation line Lc (also referred to as a "tumor cross-sectional view"), and a three-dimensional model (also referred to as a "tumor 3D model") representing the three-dimensional shape of the tumor site estimated based on the estimated infiltration distance. The infiltration distance map shown in FIG. 3(D) is a heat map in which the darker the color, the longer the infiltration distance of the tumor site. The tumor 3D model may be a graphic display (e.g., a wireframe display) used in, for example, CAD (Computer Aided Design) or the like.

[0036] The cross-section designation line Lc may be designated after the infiltration distance map is displayed, a specific example of which will be described later.

[0037] (3-2) Functional Blocks Fig. 4 is a functional block diagram of the image processing device 1 relating to display processing based on the infiltration distance of a tumor site. The processor 11 of the image processing device 1 functionally includes an endoscopic image acquisition unit 30, a tumor determination unit 31, an infiltration distance estimation unit 32, and a display control unit 33. Note that in Fig. 4, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to this. The same applies to other functional block diagrams described later.

[0038] The endoscopic image acquisition unit 30 acquires the endoscopic images Ia captured by the endoscope 3 at predetermined intervals via the interface 13. Then, the endoscopic image acquisition unit 30 supplies the acquired endoscopic images Ia to the tumor determination unit 31 and the display control unit 33, respectively.

[0039] The tumor determination unit 31 determines whether the endoscopic image Ia supplied from the endoscopic image acquisition unit 30 is a tumor-containing image. In this case, the tumor determination unit 31 detects the endoscopic image Ia as a tumor-containing image based on, for example, at least one of a user input (i.e., an external input) and an analysis result of the endoscopic image Ia. Then, when the tumor determination unit 31 detects the endoscopic image Ia as a tumor-containing image, it supplies the detected tumor-containing image to the infiltration distance estimation unit 32.

[0040] Here, detection of a tumor-containing image based on a user input will be described. In this case, when tumor determination unit 31 detects that the still image save button has been selected based on a signal supplied from operation unit 36, it detects, as a tumor-containing image, the endoscopic image Ia displayed on display device 2 at the time of selection. In this case, tumor determination unit 31 may detect, as a tumor-containing image, the latest endoscopic image Ia supplied from endoscopic image acquisition unit 30 at the time the still image save button was selected.

[0041] Next, detection of a tumor-containing image based on the analysis results of endoscopic image Ia will be described. Tumor determination unit 31 inputs endoscopic image Ia supplied from endoscopic image acquisition unit 30 into a tumor detection model constructed with reference to tumor detection model information D1, and determines whether the input endoscopic image Ia is a tumor-containing image based on the information output by the tumor detection model when endoscopic image Ia is input. For example, the tumor detection model outputs a classification result regarding the presence or absence of a tumor site in the input endoscopic image Ia, and tumor determination unit 31 determines whether the input endoscopic image Ia is a tumor-containing image based on the classification result.

[0042] Based on the infiltration distance estimation model constructed by referencing the infiltration distance estimation model information D2, the infiltration distance estimation unit 32 estimates the infiltration distance of the tumor site of the subject indicated by the tumor-containing image supplied from the tumor determination unit 31, and supplies the estimation result to the display control unit 33. In this case, in a first example, the infiltration distance estimation unit 32 generates partial images by regularly dividing the tumor-containing image (e.g., in a grid pattern), and sequentially inputs each divided partial image to the infiltration distance estimation model, thereby outputting the infiltration distance for each partial image sequentially output by the infiltration distance estimation model to the display control unit 33. In a second example, the infiltration distance estimation unit 32 inputs the tumor-containing image to the infiltration distance estimation model, and outputs an image indicating the infiltration distance for each pixel of the tumor-containing image output by the infiltration distance estimation model to the display control unit 33.

[0043] FIG. 5A shows an overview of the infiltration distance estimation process based on the first example described above. In the example of FIG. 5A, the infiltration distance estimation unit 32 divides the tumor-containing image into a grid of seven horizontal and six vertical segments to generate a total of 42 partial images for each tumor-containing image. Each partial image is input to the tumor detection model to obtain the infiltration distance at the center of each partial image. This allows the infiltration distance estimation unit 32 to obtain the distribution (map) of infiltration distances on the tumor-containing image required for generating an infiltration distance map. Instead of dividing the tumor-containing image into a grid, the infiltration distance estimation unit 32 may allow partial images to overlap and generate partial images so that the distance between the centers of adjacent partial images is shorter than the length of the partial images. This makes it possible to obtain a more detailed distribution of infiltration distances on the tumor-containing image.

[0044] Furthermore, in cases where the infiltration distance estimation unit 32 is set not to generate and display an infiltration distance map, the infiltration distance estimation unit 32 may estimate the infiltration distance limited to positions along the specified cross-section designation line Lc and obtain the infiltration distance required for the tumor cross-section. FIG. 5(B) is a diagram showing an outline of a method for estimating the infiltration distance along the cross-section designation line Lc. In the example of FIG. 5(B), the infiltration distance estimation unit 32 sets points C1 to C5 at equal intervals on the cross-section designation line Lc and sets partial images Ip1 to Ip5, which are square regions with points C1 to C5 at their centers, respectively. The infiltration distance estimation unit 32 then inputs the partial images Ip1 to Ip5 into an infiltration distance estimation model in sequence and obtains the infiltration distances output in sequence by the infiltration distance estimation model as the infiltration distances at points C1 to C5.

[0045] The infiltration distance estimation unit 32 or the display control unit 33 may further calculate a function or the like that represents the infiltration distance at any point on the cross-section designation line Lc by interpolating the infiltration distance output by the infiltration distance estimation model using any interpolation process. This enables the infiltration distance estimation unit 32 or the display control unit 33 to accurately identify the shape of the tumor site when the cross-section designation line Lc is used as the cross-section, and to display a tumor cross-sectional view that represents a smooth shape of the tumor site. Similarly, when generating an infiltration distance map, the infiltration distance estimation unit 32 or the display control unit 33 may generate an infiltration distance map by vertically and horizontally interpolating the infiltration distance output by the infiltration distance estimation model using any interpolation process.

[0046] Referring again to FIG. 4, the display control unit 33 will be described.

[0047] The display control unit 33 generates display information Ib based on the latest endoscopic image Ia supplied from the endoscopic image acquisition unit 30 and the estimated result of the infiltration distance supplied from the display control unit 33. Then, the display control unit 33 supplies the generated display information Ib to the display device 2, thereby causing the display device 2 to display the latest endoscopic image Ia and an image based on the infiltration distance. Furthermore, when the tumor determination unit 31 detects a tumor-containing image, the display control unit 33 may cause the display device 2 to display the latest tumor-containing image in addition to or instead of the latest endoscopic image Ia.

[0048] Furthermore, when the tumor determination unit 31 detects a tumor-containing image, the display control unit 33 may accept user input specifying a cross-section designation line Lc (see FIG. 3(C)) on the tumor-containing image or the latest endoscopic image Ia, generate a tumor cross-sectional view using the cross-section designation line Lc specified by the user input as a cross-section, and display the tumor cross-sectional view on the display device 2. In this case, when at least an infiltration distance map and a tumor cross-sectional view are displayed, the display control unit 33 may accept the designation of the cross-section designation line Lc after displaying the infiltration distance map. This allows the examiner to perform the operation of designating the cross-section designation line Lc while checking the position of the tumor using the infiltration distance map, thereby preferably supporting the examiner in designating the cross-section designation line Lc. The display control unit 33 may also accept user input specifying the cross-section designation line Lc on the infiltration distance map, and generate a tumor cross-sectional view using the cross-section designation line Lc specified by the user input as a cross-section.

[0049] Furthermore, when the tumor determination unit 31 detects a tumor-containing image, the display control unit 33 may control the sound output of the sound output unit 16 to output a warning sound or voice guidance notifying the user that a tumor site has been detected. Furthermore, the display control unit 33 may control the sound output of the sound output unit 16 according to the estimated result of the infiltration distance. For example, when the tumor determination unit 31 detects a tumor-containing image, the display control unit 33 may output a sound (including a pitch or melody) according to the infiltration distance at the center of the tumor-containing image. In this case, for example, information correlating the infiltration distance at the center of the tumor-containing image with the sound to be output is stored in advance, and the display control unit 33 references this information and outputs a sound according to the infiltration distance at the center of the tumor-containing image. In this way, the display control unit 33 may output a sound according to the operation of the endoscope 3 by the examiner.

[0050] The components of the endoscopic image acquisition unit 30, the tumor determination unit 31, the infiltration distance estimation unit 32, and the display control unit 33 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any nonvolatile storage medium and installed as needed to realize the components. At least some of these components may not necessarily be realized by software programs, but may be realized by any combination of hardware, firmware, and software. At least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. At least some of the components may be configured using an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, the components may be realized by various hardware. The same applies to other embodiments described below. Furthermore, the components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0051] (3-3) Display example Next, the display control of the display device 2 executed by the display control unit 33 will be described.

[0052] Fig. 6 shows a first display example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 33 of the image processing device 1 transmits display information Ib generated based on information supplied from the other processing units 30 to 32 in the processor 11 to the display device 2, thereby causing the display screen shown in Fig. 6 to be displayed on the display device 2.

[0053] In the first display example, the display control unit 33 of the image processing device 1 receives the designation of the cross-section designation line Lc after the tumor determination unit 31 detects a tumor-containing image, and displays an image based on the estimated infiltration distance for the detected tumor-containing image on the display screen. Specifically, the display control unit 33 displays an endoscopic image 70, an infiltration distance map 71, and a tumor cross-section 72 on the display screen.

[0054] The endoscopic image 70 represents a moving image based on the latest endoscopic image Ia acquired by the endoscopic image acquisition unit 30 or a still image of the latest tumor-containing image detected by the tumor determination unit 31. Note that instead of the display example of Fig. 6, the display control unit 33 may display on the display screen both a moving image based on the latest endoscopic image Ia and a still image of the latest tumor-containing image detected by the tumor determination unit 31. The display control unit 33 also displays on the endoscopic image 70 a cross-section designation line Lc designated by a user input.

[0055] Furthermore, the display control unit 33 estimates the infiltration distance of the latest tumor-containing image based on the infiltration distance estimation model, and displays an infiltration distance map 71 showing the estimation results. Here, as an example, the display control unit 33 displays an infiltration distance map 71 showing contour lines connecting positions where the infiltration distances at predetermined intervals are the same.

[0056] Furthermore, the display control unit 33 displays a tumor cross-sectional view 72 based on the infiltration distance along the cross-section designation line Lc. Here, the tumor cross-sectional view 72 also clearly shows the mucosal layer (M), muscularis mucosae (MM), and submucosa (SM) that constitute the wall layer of the large intestine, which is the subject.

[0057] Here, a specific example of a method for generating the tumor cross-sectional image 72 will be described.

[0058] In a first example, the display control unit 33 generates a tumor cross-sectional view 72 based on the estimated depth (width) of each wall layer of the colon output by an infiltration distance estimation model trained to estimate the depth of each wall layer of the colon in addition to the infiltration distance when a partial image or a tumor-containing image is input. In this case, the display control unit 33 may interpolate the estimated depth of each wall layer along the cross-section designation line Lc for each wall layer and generate the tumor cross-sectional view 72 based on the depth of each wall layer obtained by interpolation. In a second example, if information indicating standard values ​​for the depth of each wall layer of the colon is stored in the memory 12, the display control unit 33 references this information and generates a tumor cross-sectional view 72 in which the depth of each wall layer is set to the above-mentioned standard value. Note that in the tumor cross-sectional view 72, the bottom of the submucosal layer (SM) is drawn to coincide with the bottom edge of the image.

[0059] The display control unit 33 may display a 3D tumor model on the display screen in addition to or instead of the infiltration distance map 71 and the tumor cross-sectional view 72. In this case, the display control unit 33 geometrically identifies the 3D shape of the tumor site based on the estimated infiltration distance in the tumor-containing image, and displays a 3D tumor model representing the identified 3D shape on the display screen.

[0060] Fig. 7 shows a second display example of the display screen displayed by the display device 2 during an endoscopic examination. The display control unit 33 of the image processing device 1 transmits display information Ib generated based on information supplied from the other processing units 30 to 32 in the processor 11 to the display device 2, thereby causing the display screen shown in Fig. 7 to be displayed on the display device 2.

[0061] In the second display example, when the tumor determination unit 31 detects a tumor-containing image, the display control unit 33 of the image processing device 1 generates an infiltration distance map based on the estimated infiltration distance for the detected tumor-containing image, and displays the infiltration distance map superimposed on the endoscopic image 70. Here, the infiltration distance map superimposed on the endoscopic image 70 is the same as the infiltration distance map 71 in Fig. 6, and is displayed with a predetermined transmittance so that the endoscopic image 70 can be seen.

[0062] The display controller 33 then displays a text message 75 on the display screen prompting the examiner to input a cross-section designating line Lc. The display controller 33 then accepts designation of the cross-section designating line Lc based on the examiner's operation of the input unit 14. When the display controller 33 detects a signal from the input unit 14 designating the cross-section designating line Lc, the display controller 33 identifies the cross-section designating line Lc, generates a tumor cross-section based on the cross-section designating line Lc, and displays the tumor cross-section on the display screen.

[0063] Thus, according to the second display example, the display control unit 33 can preferably accept the designation of the cross-section designating line Lc for displaying a tumor cross-section. Note that, instead of the second display example, the display control unit 33 may display the infiltration distance map and the endoscopic image 70 side by side without overlapping, and accept an input to designate the cross-section designating line Lc on the infiltration distance map or the endoscopic image 70.

[0064] (3-4) Processing flow FIG. 8 is an example of a flowchart showing an outline of the processing executed by the image processing device 1 during endoscopic examination in the first embodiment.

[0065] First, the image processing device 1 acquires an endoscopic image Ia (step S11). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope 3 via the interface 13.

[0066] Next, the image processing device 1 determines whether the endoscopic image Ia acquired in step S11 corresponds to a tumor-containing image that includes a tumor site (step S12). In this case, the image processing device 1 makes the above-mentioned determination based on information output by the tumor detection model configured based on the tumor detection model information D1 when the endoscopic image Ia is input to the tumor detection model.

[0067] If the image processing device 1 determines that the endoscopic image Ia acquired in step S11 is a tumor-containing image (step S12; Yes), it calculates the infiltration distance (step S13). In this case, the image processing device 1 acquires the infiltration distance output by the infiltration distance estimation model constructed based on the infiltration distance estimation model information D2 when the tumor-containing image or a partial image thereof is input to the infiltration distance estimation model. The image processing device 1 then displays the endoscopic image Ia acquired in step S11 and an image based on the infiltration distance calculated in step S13 on the display device 2 (step S14). In this case, the image based on the infiltration distance is, for example, an infiltration distance map, a tumor cross-sectional view, or a tumor 3D model. When displaying the tumor cross-sectional view, the image processing device 1 also accepts user input specifying a cross-section designation line Lc on the endoscopic image Ia. The image processing device 1 may also display a moving image of the endoscopic image Ia and a still image of the most recent tumor-containing image.

[0068] On the other hand, if the image processing device 1 determines that the endoscopic image Ia acquired in step S11 is not a tumor-containing image (step S12; No), it displays the endoscopic image Ia acquired in step S11 on the display device 2 (step S15).

[0069] Then, after step S14 or step S15, the image processing device 1 determines whether the endoscopic examination has ended (step S16). For example, the image processing device 1 determines that the endoscopic examination has ended when it detects a predetermined input to the input unit 14 or the operation unit 36. Then, if the image processing device 1 determines that the endoscopic examination has ended (step S16; Yes), it ends the processing of the flowchart. On the other hand, if the image processing device 1 determines that the endoscopic examination has not ended (step S16; No), it returns the processing to step S11. Then, the image processing device 1 performs the processing of steps S11 to S15 on the endoscopic image Ia newly generated by the endoscope 3.

[0070] (4) Variations Next, preferred modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

[0071] (Variation 1) When displaying an infiltration distance map, the image processing device 1 may generate an infiltration distance map for a portion of the tumor-containing image instead of generating an infiltration distance map for the entire tumor-containing image.

[0072] For example, the image processing device 1 generates and displays an invasion distance map targeting a rectangular area including a cross-section designation line Lc designated by the examiner (e.g., the smallest rectangular area including the cross-section designation line Lc). In this case, the image processing device 1 identifies the cross-section designation line Lc based on user input specifying the cross-section designation line Lc, and then displays an invasion distance map and a tumor cross-sectional image. In this embodiment, the image processing device 1 can display an invasion distance map limited to a region of interest to the examiner.

[0073] (Variation 2) The image processing device 1 may automatically set the cross-section designating line Lc instead of displaying the cross-section designating line Lc designated based on a user input.

[0074] In this case, for example, the image processing device 1 generates an infiltration distance map for the entire tumor-containing image and sets a cross-section designation line Lc of a predetermined length that passes through at least the point with the longest infiltration distance on the infiltration distance map. In another example, the image processing device 1 approximates the region where the infiltration distance is equal to or greater than a predetermined distance with an ellipse and sets a cross-section designation line Lc corresponding to the major axis of the approximated ellipse. According to this embodiment, the image processing device 1 can display a tumor cross-section of the tumor site without relying on input from the examiner.

[0075] (Variation 3) The image processing device 1 may process a video image composed of endoscopic images Ia generated during an endoscopic examination after the examination.

[0076] For example, at any timing after an examination, when an image to be processed is designated based on user input via the input unit 14, the image processing device 1 sequentially performs the processing of the flowchart in Fig. 8 on the time-series endoscopic images Ia constituting the designated image. Then, when it is determined in step S16 that the target image has ended, the image processing device 1 ends the processing of the flowchart, and when the target image has not ended, the image processing device 1 returns to step S11 and performs the processing of the flowchart on the next endoscopic image Ia in the time series.

[0077] (Variation 4) The tumor detection model information D1 and the infiltration distance estimation model information D2 may be stored in a storage device separate from the image processing device 1.

[0078] 9 is a schematic configuration diagram of an endoscopic examination system 100A in Modification 3. For simplicity, the display device 2 and the endoscope 3 are not shown. The endoscopic examination system 100A includes a server device 4 that stores tumor detection model information D1 and infiltration distance estimation model information D2. The endoscopic examination system 100A also includes a plurality of image processing devices 1 (1A, 1B, ...) that are capable of data communication with the server device 4 via a network.

[0079] In this case, each image processing device 1 references tumor detection model information D1 and infiltration distance estimation model information D2 via the network. In this case, interface 13 of each image processing device 1 includes a communication interface such as a network adapter for communication. In this configuration, each image processing device 1 can reference tumor detection model information D1 and infiltration distance estimation model information D2, as in the above-described embodiment, and preferably executes processing related to lesion detection.

[0080] Second Embodiment 10 is a block diagram of an image processing device 1X according to the second embodiment. The image processing device 1X includes an infiltration distance acquisition unit 32X and an output control unit 33X. The image processing device 1X may be composed of multiple devices.

[0081] The infiltration distance acquisition means 32X acquires the infiltration distance of the tumor site of the subject in the endoscopic image based on an endoscopic image of the subject captured by an imaging unit provided in the endoscope. The infiltration distance acquisition means 32X can be, for example, the infiltration distance estimation unit 32 in the first embodiment (including modified examples, the same applies below). In another example, the infiltration distance acquisition means 32X may acquire the infiltration distance by receiving an estimation result of the infiltration distance of the tumor site of the subject from an external device (i.e., a device separate from the image processing device 1X) that executes processing equivalent to the infiltration distance estimation unit 32 in the first embodiment.

[0082] The output control means 33X outputs an image or sound based on the infiltration distance to an output device. The output control means 33X can be the display control unit 33 in the first embodiment. The output device can be at least one of the display device 2 and the sound output unit 16 in the first embodiment. The output device may also be incorporated into the image processing device 1X.

[0083] 11 is an example of a flowchart showing a processing procedure in the second embodiment. The infiltration distance acquisition means 32X acquires the infiltration distance of the tumor site in the subject in the endoscopic image based on the endoscopic image of the subject captured by the imaging unit provided in the endoscope (step S21). The output control means 33X outputs an image or sound based on the infiltration distance to the output device (step S22).

[0084] According to the second embodiment, the image processing device 1X can present information on the infiltration distance of a tumor site in a subject, which information is included in an endoscopic image of the subject.

[0085] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0086] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0087] [Appendix 1] an infiltration distance acquisition means for acquiring an infiltration distance of a tumor site in an endoscope image based on an endoscope image of the subject captured by an imaging unit provided in the endoscope; an output control means for outputting an image or sound based on the infiltration distance to an output device; An image processing device having: [Appendix 2] 2. The image processing device according to claim 1, wherein the output control means displays, on the output device, a map of the infiltration distance in the endoscopic image including the tumor site as an image based on the infiltration distance. [Appendix 3] 3. The image processing device according to claim 2, wherein the map is a contour map of the infiltration distance or a heat map of the infiltration distance other than a contour map. [Appendix 4] 2. The image processing device according to claim 1, wherein the output control means displays, on the output device, a cross-sectional view of the subject at the tumor site as an image based on the infiltration distance. [Appendix 5] 5. The image processing device of claim 4, wherein the cross-sectional view includes the tumor site and a wall layer of the subject. [Appendix 6] 6. The image processing device according to claim 4, wherein the output control means displays on the output device the cross-sectional view, with a specified line on the endoscopic image displayed on the output device as a cutting plane. [Appendix 7] 7. The image processing device according to claim 6, wherein the output control means displays a map of the infiltration distance in the endoscopic image including the tumor site on the output device, and then accepts external input specifying the line. [Appendix 8] 2. The image processing device according to claim 1, wherein the output control means displays a three-dimensional model of the tumor site on the output device as an image based on the infiltration distance. [Appendix 9] The imaging device further includes a tumor determination means for determining an endoscopic image including the tumor site from the endoscopic image captured by the imaging unit, 2. The image processing device according to claim 1, wherein the infiltration distance acquisition means estimates the infiltration distance based on the endoscopic image determined to include the tumor site. [Appendix 10] the infiltration distance acquisition means estimates the infiltration distance based on a model to which the endoscopic image determined to include the tumor site or a partial image of the endoscopic image is input; 10. The image processing device according to claim 9, wherein the model is a model that has learned a relationship between an image input to the model and the infiltration distance of the subject depicted in the image. [Appendix 11] The computer acquiring an infiltration distance of a tumor site in an endoscope image based on an endoscope image of the subject captured by an imaging unit provided in the endoscope; outputting an image or sound based on the infiltration distance to an output device; Image processing methods. [Appendix 12] acquiring an infiltration distance of a tumor site in an endoscope image based on an endoscope image of the subject captured by an imaging unit provided in the endoscope; A storage medium storing a program that causes a computer to execute a process of outputting an image or sound based on the infiltration distance to an output device.

[0088] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent documents and other documents are incorporated herein by reference. [Explanation of symbols]

[0089] 1, 1A, 1B, 1X Image Processing Device 2 Display device 3 Endoscope 11 processors 12 Memory 13 Interface 14 Input section 15 Light source section 16 Sound output section 100, 100A Endoscopy System

Claims

1. an estimation means for estimating, based on an endoscopic image of a subject photographed by an imaging unit provided in an endoscope, the invasion distance of a tumor site in the subject at a predetermined cross section in the endoscopic image and the depth of each wall layer of the subject at the cross section; an output control means for displaying, on an output device, a cross-sectional view of the subject at the cutting plane showing the tumor site and each of the wall layers based on the estimated invasion distance and the estimated depth of each of the wall layers; An image processing device having:

2. The image processing device according to claim 1 , wherein the output control means displays a map of the infiltration distance in the endoscopic image including the tumor site on the output device.

3. The image processing device according to claim 2 , wherein the map is a contour map of the infiltration distance or a heat map of the infiltration distance other than a contour map.

4. The image processing apparatus according to claim 1 , wherein said output control means displays on said output device said cross-sectional view, the cross-section of which is taken along a line specified on said endoscopic image displayed on said output device.

5. The image processing device according to claim 4 , wherein the output control means receives an external input specifying the line after displaying a map of the infiltration distance in the endoscopic image including the tumor site on the output device.

6. The image processing device according to claim 1 , wherein the output control means displays a three-dimensional model of the tumor site on the output device.

7. The imaging device further includes a tumor determination means for determining an endoscopic image including the tumor site from the endoscopic image captured by the imaging unit, The image processing device according to claim 1 , wherein the estimation means estimates the infiltration distance based on the endoscopic image determined to include the tumor site.

8. the estimation means estimates the infiltration distance based on a model to which the endoscopic image determined to include the tumor site or a partial image of the endoscopic image is input; The image processing apparatus according to claim 7 , wherein the model is a model that has learned a relationship between an image input to the model and the infiltration distance of the subject depicted in the image.

9. The computer Based on an endoscopic image of a subject photographed by an imaging unit provided in an endoscope, an infiltration distance of a tumor site in the subject at a predetermined cross section in the endoscopic image and a depth of each wall layer of the subject at the cross section are estimated; displaying, on an output device, a cross-sectional view of the subject along the cutting plane, showing the tumor site and each of the wall layers, based on the estimated invasion distance and the estimated depth of each of the wall layers; Image processing methods.

10. Based on an endoscopic image of a subject photographed by an imaging unit provided in an endoscope, an infiltration distance of a tumor site in the subject at a predetermined cross section in the endoscopic image and a depth of each wall layer of the subject at the cross section are estimated; A program that causes a computer to execute a process of displaying on an output device a cross-sectional view of the subject at the cut surface showing the tumor site and each of the wall layers based on the estimated infiltration distance and the estimated depth of each of the wall layers.

Images