Image processing device, image processing method, and program

Abstract

An image processing device 1X is mainly provided with an acquisition means 30X, a lesion part detection means 31X, and a lesion diagnosis means 32X. The acquisition means 30X acquires a captured image obtained by capturing, by an image capturing unit provided in an endoscope, an image of a subject to be examined. The lesion part detection means 31X detects a lesion part that is suspected as a lesion in the captured image. The lesion diagnostic means 32X makes a diagnosis concerning the lesion part on the basis of the captured image, and a cropped image in which the lesion part is cropped from the captured image.

Description

[Technical field]

[0001] The present disclosure relates to the technical field of an image processing device, an image processing method, and a storage medium that process images acquired in an endoscopic examination. [Background technology]

[0002] Conventionally, there has been known an endoscope system that displays an image of the inside of an organ lumen. For example, Patent Document 1 discloses an endoscope system that detects a lesion in an image captured by an endoscope based on hue, size, and the like, and highlights the detected lesion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-135983 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, after detecting a lesion, the lesion is highlighted, but it is left to the examiner to specifically diagnose the condition of the detected lesion. However, depending on the lesion, it may be difficult for the examiner to diagnose the lesion.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of suitably performing diagnosis of a lesion site in an endoscopic examination. [Means for solving the problem]

[0006] One aspect of the image processing device is An acquisition means for acquiring an image of an object to be examined by an imaging unit provided in the endoscope; a lesion site detection means for detecting a lesion site suspected of being a lesion in the photographed image; A diagnostic model outputs a diagnosis result regarding a lesion site in an image when the image is input, and outputs the diagnosis result by inputting an image obtained by combining the photographed image and a cut-out image obtained by cutting out the lesion site from the photographed image. Get A diagnostic tool; an output control means for outputting information based on the diagnosis result to a display device or a sound output device; The image processing device has the following features.

[0007] One aspect of the image processing method includes: The computer An image of the object to be examined is obtained by an imaging unit provided in the endoscope; Detecting a lesion site suspected of being a lesion in the captured image; A diagnostic model outputs a diagnosis result regarding a lesion site in an image when the image is input, and outputs the diagnosis result by inputting an image obtained by combining the photographed image and a cut-out image obtained by cutting out the lesion site from the photographed image. Get , outputting information based on the diagnosis result to a display device or a sound output device; An image processing method.

[0008] One aspect of the program is An image of the object to be examined is obtained by an imaging unit provided in the endoscope; Detecting a lesion site suspected of being a lesion in the captured image; A diagnostic model outputs a diagnosis result regarding a lesion site in an image when the image is input, and outputs the diagnosis result by inputting an image obtained by combining the photographed image and a cut-out image obtained by cutting out the lesion site from the photographed image. Get , Outputting information based on the diagnosis result to a display device or a sound output device It is a program that causes a computer to execute processing. Effect of the Invention

[0009] Diagnosis of the lesion site can be suitably performed from the endoscopic image. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a schematic configuration of an endoscopic examination system. [Diagram 2] 2 shows the hardware configuration of an image processing device. [Diagram 3] FIG. 2 is a diagram illustrating an outline of a process executed by the image processing apparatus. [Figure 4] FIG. 2 is a functional block diagram of the image processing device. [Diagram 5] (A) shows a first embodiment of generating an input image to be input to a lesion diagnosis model, and (B) shows a second embodiment of generating an input image to be input to a lesion diagnosis model. [Figure 6] 4 shows an example of a display screen displayed by a display device during an endoscopic examination. [Figure 7] 4 is an example of a flowchart illustrating an overview of processing by the image processing device in the first embodiment. [Figure 8] FIG. 13 is a schematic configuration diagram of an endoscopic inspection system according to a modified example. [Figure 9] FIG. 11 is a block diagram of an image processing device according to a second embodiment. [Figure 10] 13 is an example of a flowchart executed by the image processing device in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. The endoscopic examination system 100 detects a site suspected of being a lesion (lesion site) for an examiner such as a doctor who performs an examination (including treatment) using an endoscope, and performs a diagnosis on the detected lesion site. As shown in Fig. 1, the endoscopic examination system 100 mainly includes an image processing device 1, a display device 2, and an endoscope scope 3 connected to the image processing device 1.

[0013] The image processing device 1 acquires images (also referred to as "captured images Ia") captured by the endoscope 3 in time series from the endoscope 3, and displays a screen based on the captured images Ia on the display device 2. The captured images Ia are images captured at a predetermined time interval during at least one of the steps of inserting or ejecting the endoscope 3 into the subject. In this embodiment, the image processing device 1 analyzes the captured images Ia to detect a lesion site and diagnose the lesion site, and displays information related to the diagnosis result on the display device 2.

[0014] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0015] The endoscope scope 3 mainly comprises an operation unit 36 ​​through which the examiner makes predetermined inputs, a flexible shaft 37 that is inserted into the subject's organ to be imaged, a tip portion 38 that incorporates an imaging unit such as an ultra-miniature imaging element, and a connection portion 39 for connecting to the image processing device 1.

[0016] In the following, the process of endoscopic examination of the large intestine will be described as a representative example, but the object of examination is not limited to the large intestine, and may be the digestive tract (digestive organs) such as the stomach, esophagus, small intestine, and duodenum. In addition, the image processing device 1 in this embodiment can accurately perform a diagnosis to identify lesions such as hyperplastic polyps, serrated lesions, adenomas, and intraepithelial carcinomas that are difficult to diagnose without considering the difference in the surface pattern between normal areas and lesions, using a method to be described later. These lesions are formed not only in the large intestine but also in at least the stomach. Therefore, when at least the large intestine and stomach are examined, the endoscopy examination system 100 can accurately perform a diagnosis to identify the above-mentioned lesions and appropriately support the examiner. In addition, the image processing device 1 can perform a diagnosis to identify any other lesion, not limited to a diagnosis to identify a lesion that is difficult to diagnose without considering the difference in the surface pattern between normal areas and lesions.

[0017] (2) Hardware Configuration 2 shows a hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0018] The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device, such as a hard disk connected to or built into the image processing device 1, or may include a removable storage medium, such as a flash memory. The memory 12 stores programs for the image processing device 1 to execute each process in this embodiment.

[0020] The memory 12 also stores lesion detection model information D1 and lesion diagnosis model information D2.

[0021] The lesion detection model information D1 is information about a lesion detection model that detects a lesion site from an input image. The lesion detection model information D1 includes parameters necessary for configuring the lesion detection model. The lesion detection model is, for example, a machine learning model such as a neural network or a support vector machine, and is configured to output a detection result of a lesion site (site with abnormality) in the input photographed image Ia when the photographed image Ia is input. Representative models of neural networks used in such lesion detection models include, for example, a Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. The detection result output by the lesion detection model may be a mask image (a binary image in which the values ​​of the pixels of the lesion site and the other pixels are different) showing an area in the photographed image Ia corresponding to the lesion site, or may be a reliability map that is a map on an image showing the reliability of the lesion site. Here, the reliability is an index indicating the degree of possibility that the site is a lesion site, and the higher the reliability, the higher the possibility that the site is a lesion site. In yet another example, the lesion detection model may be a model that outputs information indicating a rectangular area (bounding box) of a lesion site when the photographed image Ia is input. Based on the detection result output by the lesion detection model, the image processing device 1 generates an image (also referred to as a "lesion cropped image Ic") obtained by cutting out (cropping) an area including the lesion site detected from the photographed image Ia.

[0022] The lesion diagnosis model information D2 is information about a lesion diagnosis model that is a model for diagnosing a lesion site shown in an input image. The lesion diagnosis model information D2 includes parameters required to configure the lesion diagnosis model. The lesion diagnosis model is, for example, a machine learning model such as a neural network or a support vector machine, and is configured to output a diagnosis result of the lesion site shown by the lesion cut-out image Ic when the lesion cut-out image Ic and the corresponding photographed image Ia are input simultaneously (i.e., input together as one input image). The diagnosis result output by the lesion diagnosis model is, for example, classification information indicating the name of a disease or the type (category) of a lesion corresponding to the target lesion site. This classification information may include a classification result regarding the risk of the target lesion site becoming cancerous or the like.

[0023] The lesion detection model and the lesion diagnosis model are trained in advance based on a pair of an input image conforming to the input format of each model and correct answer data indicating the correct answer to be output when the input image is input to each model. Then, the parameters of each model obtained by training are stored in memory 12 as lesion detection model information D1 and lesion diagnosis model information D2, respectively.

[0024] At least one of the lesion detection model information D1 and the lesion diagnosis model information D2 may be stored in an external device capable of data communication with the image processing device 1 via wired or wireless communication, instead of the memory 12.

[0025] The interface 13 performs an interface operation between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the captured image Ia supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for communicating with an external device by wire or wirelessly, or may be a hardware interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0026] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, a voice input device, etc. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also include a built-in pump for sending out water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0027] (3) Image processing device processing FIG. 3 is a diagram showing an outline of the process executed by the image processing device 1. As shown in FIG.

[0028] First, the image processing device 1 generates a lesion cut-out image Ic by cutting out (cropping) an area corresponding to a lesion site from the photographed image Ia based on a lesion detection model configured based on the lesion detection model information D1. The image processing device 1 also generates an image (also called an "entire resized image Id") obtained by resizing the photographed image Ia used to cut out the lesion cut-out image Ic in accordance with the input format of the lesion diagnosis model. In this case, the image processing device 1 performs a process of converting the photographed image Ia into the entire resized image Id using an arbitrary image resizing method.

[0029] Next, when the pair of the lesion cut-out image Ic and the resized entire image Id is obtained, the image processing device 1 simultaneously inputs the lesion cut-out image Ic and the resized entire image Id to a lesion diagnosis model configured based on the lesion diagnosis model information D2 to obtain a diagnosis result for the lesion site indicated by the lesion cut-out image Ic. In this case, the image processing device 1 generates an input image to the lesion diagnosis model that combines the lesion cut-out image Ic and the resized entire image Id by superimposing the lesion cut-out image Ic and the resized entire image Id in the channel direction, for example. Then, the image processing device 1 causes the display device 2 to display information related to the diagnosis result.

[0030] By performing such processing, the image processing device 1 can accurately perform diagnosis to identify each lesion, such as hyperplastic polyps, serrated lesions, adenomas, and intraepithelial carcinomas, which are difficult to diagnose without taking into account the differences in the surface patterns of normal areas and lesions, and present the diagnostic results to the examiner.

[0031] Fig. 4 is a functional block diagram of the image processing device 1. As shown in Fig. 4, the processor 11 of the image processing device 1 functionally includes a photographed image acquisition section 30, a lesion site detection section 31, a lesion diagnosis section 32, and a display control section 33. Note that in Fig. 4, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to that shown in Fig. 4. The same applies to other functional block diagrams described later.

[0032] The photographed image acquisition unit 30 acquires the photographed images Ia photographed by the endoscope 3 at predetermined intervals via the interface 13. Then, the photographed image acquisition unit 30 supplies the acquired photographed images Ia to the lesion site detection unit 31, the lesion diagnosis unit 32, and the display control unit 33, respectively.

[0033] Lesion site detection unit 31 detects a lesion site based on photographed image Ia. In this case, lesion site detection unit 31 inputs photographed image Ia to a lesion detection model configured based on lesion detection model information D1, and acquires a detection result output by the lesion detection model. Then, when lesion site detection unit 31 determines that a lesion site is present in photographed image Ia based on the acquired detection result, it generates a lesion cut-out image Ic based on the detection result, and supplies lesion cut-out image Ic to lesion diagnosis unit 32.

[0034] Here, a supplementary explanation will be given on the manner of generating the lesion cut-out image Ic based on the detection result output by the lesion detection model. For example, when the detection result output by the lesion detection model indicates a pixel of the lesion site, the lesion site detection unit 31 generates the lesion cut-out image Ic by cutting out the smallest rectangular area including the pixel from the photographed image Ia. In another example, when the detection result output by the lesion detection model indicates the reliability of the lesion site for each pixel, the lesion site detection unit 31 generates the lesion cut-out image Ic by cutting out the smallest rectangular area including a pixel whose reliability is equal to or greater than a predetermined threshold from the photographed image Ia. In yet another example, when the detection result output by the lesion detection model indicates a rectangular area (bounding box) of the lesion site, the lesion site detection unit 31 generates the lesion cut-out image Ic by cutting out the rectangular area from the photographed image Ia.

[0035] The lesion diagnosis unit 32 performs a diagnosis on the lesion site detected by the lesion site detection unit 31. In this case, first, the lesion diagnosis unit 32 acquires the lesion cut-out image Ic supplied from the lesion site detection unit 31 and the captured image Ia from which the lesion cut-out image Ic is cut out. Next, the lesion diagnosis unit 32 generates an entire resized image Id by resizing the acquired captured image Ia based on the input format of the lesion diagnosis model. Then, the lesion diagnosis unit 32 inputs an image in which the lesion cut-out image Ic and the entire resized image Id are combined as one input image to the lesion diagnosis model configured based on the lesion diagnosis model information D2. Then, the lesion diagnosis unit 32 acquires the diagnosis result output by the lesion diagnosis model. In this way, the lesion diagnosis unit 32 performs a diagnosis on the lesion site based on the diagnosis result output by the lesion diagnosis model. Furthermore, the lesion diagnosis unit 32 supplies the diagnosis result to the display control unit 33.

[0036] The lesion diagnosis unit 32 may directly supply the diagnosis result output by the lesion diagnosis model to the display control unit 33, or may supply the diagnosis result enhanced based on prior knowledge information on the classification of the lesion stored in the memory 12 or the like to the display control unit 33. The prior knowledge information includes, for example, at least one of information associating the classification of the lesion with the risk of the lesion, such as cancer, and information associating the classification of the lesion with the name of a disease corresponding to the lesion. For example, when the information associating the classification of the lesion with the risk of the lesion, such as cancer, is stored in the memory 12 as prior knowledge information, the lesion diagnosis unit 32 identifies the corresponding risk based on the classification of the lesion output by the lesion diagnosis model as a diagnosis result, with reference to the prior knowledge information. Then, the lesion diagnosis unit 32 may supply the classification information of the lesion output by the lesion diagnosis model and information on the risk of the lesion based on the prior knowledge information to the display control unit 33 as a diagnosis result. In addition, when information associating a lesion classification with a disease name corresponding to the lesion is stored in memory 12 as prior knowledge information, lesion diagnosis unit 32 may refer to the prior knowledge information and supply a diagnosis result including the disease name to display control unit 33.

[0037] Here, the process of generating a resized entire image Id by resizing the acquired photographed image Ia based on the input format of the lesion diagnosis model, and the process of combining the lesion cut-out image Ic and the resized entire image Id into one input image are not essential processes of the lesion diagnosis unit 32, and at least one of these processes may be executed by a processing block (including a processing block not shown in FIG. 4) other than the lesion diagnosis unit 32. For example, when the process of generating the resized entire image Id is executed by another processing block, the lesion diagnosis unit 32 acquires the resized entire image Id, which is the photographed image Ia resized based on the input format of the lesion diagnosis model, from the other processing block. Also, when the process of combining the lesion cut-out image Ic and the resized entire image Id into one input image is executed by another processing block, the lesion diagnosis unit 32 acquires the image in which the lesion cut-out image Ic and the resized entire image Id are combined from the other processing block as an input image of the lesion diagnosis model.

[0038] The display control unit 33 generates display information Ib based on the photographed image Ia, the detection result of the lesion site by the lesion detection unit 31, and the diagnosis result of the lesion site by the lesion diagnosis unit 32, and supplies the display information Ib to the display device 2 via the interface 13. A specific example of the display control of the display device 2 by the display control unit 33 will be described later.

[0039] Here, each of the components of the photographed image acquisition unit 30, the lesion site detection unit 31, the lesion diagnosis unit 32, and the display control unit 33 can be realized, for example, by the processor 11 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. At least a part of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of these components may be realized by using an integrated circuit that is programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program composed of each of the above components. Also, at least a part of each component may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by cooperation of multiple computers, for example, using cloud computing technology.

[0040] Next, a supplementary explanation will be given on the generation process of the input image to be input to the lesion diagnosis model. Fig. 5(A) shows a first mode of generation of the input image to be input to the lesion diagnosis model, and Fig. 5(B) shows a second mode of generation of the input image to be input to the lesion diagnosis model.

[0041] In the first mode, the lesion diagnosis unit 32 generates an input image for the lesion diagnosis model by superimposing, in the channel direction, the lesion cut-out image Ic and the entire resized image Id (both of which have a horizontal size of "X", a vertical size of "Y", and a channel count of "Z"), whose vertical and horizontal sizes have been adjusted to fit the input format of the lesion diagnosis model. In this case, the generated input image is an image with a horizontal size of "X", a vertical size of "Y", and a channel count of "Z+Z". For example, when the captured image Ia is an RGB image (i.e., Z=3), the lesion diagnosis unit 32 generates an image with a channel count of 6 (i.e., Z=6) by superimposing, in the channel direction, the lesion cut-out image Ic and the entire resized image Id, as the input image for the lesion diagnosis model.

[0042] In the second embodiment, the lesion diagnosis unit 32 generates an input image for the lesion diagnosis model by combining the lesion cut-out image Ic and the entire resized image Id vertically or horizontally. In the example of FIG. 5(B), the lesion diagnosis unit 32 generates an input image for the lesion diagnosis model by vertically overlapping the lesion cut-out image Ic and the entire resized image Id (both here being images with a horizontal size of "X", a vertical size of "Y", and a number of channels of "Z"). In this case, the generated input image is an image with a horizontal size of "X", a vertical size of "Y+Y", and a number of channels of "Z". Note that the lesion cut-out image Ic and the entire resized image Id are adjusted so that at least their horizontal sizes are the same when combined vertically, and are adjusted so that at least their vertical sizes are the same when combined horizontally.

[0043] By generating the input image exemplified by the first or second embodiment, the lesion diagnosis unit 32 can comprehensively identify the classification of the lesion by considering both the periphery of the lesion and the entire image. This enables the image processing device 1 to perform highly accurate diagnosis of a lesion site that is difficult to diagnose without considering the difference in surface patterns between the normal region and the lesion.

[0044] Note that the generation of the input image exemplified by the first or second aspect is not essential processing of the lesion diagnosis unit 32, and may be performed by another processing block. For example, when another processing block performs processing to superimpose the lesion cut-out image Ic and the resized entire image Id in the channel direction, the lesion diagnosis unit 32 acquires the image in which the lesion cut-out image Ic and the resized entire image Id are superimposed in the channel direction from the other processing block as an input image to the lesion diagnosis model. Also, when another processing block performs processing to superimpose the lesion cut-out image Ic and the resized entire image Id in the vertical or horizontal direction, the lesion diagnosis unit 32 acquires the image in which the lesion cut-out image Ic and the resized entire image Id are superimposed in the vertical or horizontal direction from the other processing block as an input image to the lesion diagnosis model.

[0045] Next, the display control of the display device 2 executed by the display control unit 33 will be described.

[0046] Fig. 6 shows an example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 33 of the image processing device 1 transmits to the display device 2 display information Ib generated based on the captured image Ia acquired by the captured image acquisition unit 30, the detection result by the lesion site detection unit 31, and the diagnosis result by the lesion diagnosis unit 32, thereby causing the display device 2 to display the display screen shown in Fig. 6.

[0047] In the example of Fig. 6, the display control unit 33 of the image processing device 1 displays the latest captured image Ia (i.e., moving images switched at a predetermined drawing rate) acquired by the captured image acquisition unit 30 on the display screen. Furthermore, the display control unit 33 displays a comment 72 that mentions that a lesion site classified as a serrated lesion has been detected based on the diagnosis result supplied from the lesion diagnosis unit 32, and that the serrated lesion may become cancerous. In this way, the comment 72 corresponds to information on the classification and risk of the lesion site. Note that the display control unit 33 may display a comment on either the classification or risk of the lesion site instead of the comment 72.

[0048] Furthermore, based on the detection results supplied from the lesion site detection unit 31, the display control unit 33 displays, together with the above-mentioned comment 72, a rectangular frame 71 that highlights a rectangular area corresponding to the lesion cut-out image Ic on the captured image Ia.

[0049] Furthermore, when the diagnosis result supplied from the lesion diagnosing unit 32 indicates a specific type of lesion (e.g., a malignant lesion), the display control unit 33 may output information notifying the examiner of the presence of the specific type of lesion. In this case, for example, the display control unit 33 may highlight the captured image Ia, etc. being displayed on the display screen by using a border effect, and may output a warning sound, a guidance voice, etc. from the sound output unit 16.

[0050] In this way, when a lesion site is detected, the display control unit 33 clearly indicates the lesion site and displays the diagnosis result together with the latest captured image Ia, thereby preferably assisting the examiner in grasping the lesion site. Note that the output mode of the diagnosis result and the like by the display control unit 33 is not limited to the example shown in Fig. 7, and various modes may be used. For example, instead of displaying the comment 72, the display control unit 33 may output a sound corresponding to the comment 72 from the sound output unit 16.

[0051] FIG. 7 is an example of a flowchart showing an overview of the processing executed by the image processing device 1 when an endoscopic examination is started in the first embodiment.

[0052] First, the photographed image acquisition section 30 of the image processing device 1 acquires the photographed image Ia (step S11). In this case, the photographed image acquisition section 30 of the image processing device 1 receives the photographed image Ia from the endoscope scope 3 via the interface 13.

[0053] Next, lesion site detection unit 31 of image processing device 1 detects a lesion site in photographed image Ia (step S12). In this case, lesion site detection unit 31 inputs photographed image Ia to a lesion detection model configured based on lesion detection model information D1, thereby acquiring a detection result of a lesion site in photographed image Ia.

[0054] Then, the lesion site detection unit 31 determines whether or not a lesion site is detected (step S13). If the lesion site detection unit 31 does not detect a lesion site (step S13; No), the lesion site detection unit 31 causes the display device 2 to display the captured image Ia acquired in step S11 (step S18).

[0055] On the other hand, if the lesion site detection unit 31 detects a lesion site (step S13; Yes), it cuts out an area including the detected lesion site from the photographed image Ia (step S14), thereby generating a lesion cut-out image Ic.

[0056] Then, the lesion diagnosis unit 32 inputs the whole resized image Id obtained by resizing the photographed image Ia acquired in step S11 and the lesion cut-out image Ic to the lesion diagnosis model at the same time (step S15). In this case, the lesion diagnosis unit 32 inputs an input image generated in the manner shown in either FIG. 5(A) or FIG. 5(B), for example, to the lesion diagnosis model configured based on the lesion diagnosis model information D2. Then, the lesion diagnosis unit 32 acquires a diagnosis result based on the output of the lesion diagnosis model (step S16). Then, the display control unit 33 causes the display device 2 to display the latest photographed image Ia and the diagnosis result acquired in step S16 (step S17).

[0057] After step S17 or step S18, the image processing device 1 determines whether or not the endoscopic examination has ended (step S19). For example, when the image processing device 1 detects a predetermined input to the input unit 14 or the operation unit 36, the image processing device 1 determines that the endoscopic examination has ended. Then, when the image processing device 1 determines that the endoscopic examination has ended (step S19; Yes), the image processing device 1 ends the processing of the flowchart. On the other hand, when the image processing device 1 determines that the endoscopic examination has not ended (step S19; No), the image processing device 1 returns the processing to step S11. Then, the image processing device 1 executes the processing of steps S11 to S19 on the captured image Ia newly generated by the endoscope 3.

[0058] (4) Variations Next, preferred modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

[0059] (Variation 1) The image processing device 1 may process a video image constructed from the captured images Ia generated during an endoscopic examination after the examination.

[0060] For example, at any timing after an examination, when an image to be processed is designated based on user input via the input unit 14, the image processing device 1 sequentially performs the process of the flowchart in Fig. 7 on the time-series captured images Ia constituting the designated image. Then, when it is determined in step S19 that the target image has ended, the image processing device 1 ends the process of the flowchart, and when the target image has not ended, the image processing device 1 returns to step S11 and performs the process of the flowchart on the next captured image Ia in the time series.

[0061] (Variation 2) The lesion detection model information D1 and the lesion diagnosis model information D2 may be stored in a storage device separate from the image processing device 1.

[0062] Fig. 8 is a schematic configuration diagram of an endoscopic examination system 100A in Modification 2. For simplicity, Fig. 8 does not show the display device 2 and the endoscope 3. The endoscopic examination system 100A shown in Fig. 8 includes a server device 4 that stores lesion detection model information D1 and lesion diagnosis model information D2. The endoscopic examination system 100A also includes a plurality of image processing devices 1 (1A, 1B, ...) that are capable of data communication with the server device 4 via a network.

[0063] In this case, each image processing device 1 refers to lesion detection model information D1 and lesion diagnosis model information D2 via the network. In this case, interface 13 of each image processing device 1 includes a communication interface such as a network adapter for performing communications. In this configuration, each image processing device 1 can refer to lesion detection model information D1 and lesion diagnosis model information D2, as in the above-described embodiment, and suitably execute detection / diagnosis processing and display processing of a lesion site.

[0064] <Second embodiment> 9 is a block diagram of an image processing device 1X in the second embodiment. The image processing device 1X mainly includes an acquisition unit 30X, a lesion site detection unit 31X, and a lesion diagnosis unit 32X. Note that the image processing device 1X may be composed of multiple devices.

[0065] The acquiring means 30X acquires a photographed image of an object to be inspected by an imaging unit provided in the endoscope. The acquiring means 30X can be the photographed image acquiring unit 30 in the first embodiment (including modified examples, the same applies below). The acquiring means 30X may instantly acquire a photographed image generated by the imaging unit, or may acquire a photographed image generated in advance by the imaging unit and stored in a storage device at a predetermined timing.

[0066] The lesion detection means 31X detects a lesion site suspected of being a lesion in the captured image. The lesion detection means 31X can be the lesion detection section 31 in the first embodiment.

[0067] Lesion diagnosing means 32X performs diagnosis of the lesion site based on the captured image and a cut-out image obtained by cutting out the lesion site from the captured image. Lesion diagnosing means 32X can be lesion diagnosing section 32 in the first embodiment.

[0068] 10 is an example of a flowchart showing a processing procedure in the second embodiment. First, the acquisition means 30X acquires a captured image of an object to be examined by an imaging unit provided in the endoscope (step S21). Next, the lesion site detection means 31X detects a lesion site suspected to be a lesion in the captured image (step S22). Then, the lesion diagnosis means 32X performs a diagnosis regarding the lesion site based on the captured image and a cut-out image obtained by cutting out the lesion site from the captured image (step S23).

[0069] According to the second embodiment, the image processing device 1X can accurately diagnose lesions that are difficult to diagnose without taking into account the difference in surface patterns between normal areas and lesions, based on a captured image showing a wide area of ​​the test subject and an image of a cut-out area suspected of being a lesion.

[0070] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0071] [Appendix 1] An acquisition means for acquiring an image of an object to be examined by an imaging unit provided in the endoscope; a lesion site detection means for detecting a lesion site suspected of being a lesion in the photographed image; a diagnostic means for diagnosing the lesion site based on the photographed image and a cut-out image of the lesion site cut out from the photographed image; An image processing device comprising: [Appendix 2] The diagnostic means inputs an image obtained by combining the captured image and the cropped image into a diagnostic model that outputs a diagnostic result regarding the lesion area in the image when the image is input, and performs the diagnosis based on the diagnostic result output by the diagnostic model. [Appendix 3] The image processing device described in Appendix 2, wherein the diagnostic means inputs an image in which the captured image and the cropped image are superimposed in the channel direction into the diagnostic model, and performs the diagnosis based on the diagnostic result output by the diagnostic model. [Appendix 4] The image processing device described in Appendix 2, wherein the diagnostic means inputs an image in which the captured image and the cropped image are combined vertically or horizontally into the diagnostic model, and performs the diagnosis based on the diagnostic result output by the diagnostic model. [Appendix 5] The image processing device described in any one of Appendices 2 to 4, wherein the diagnostic means inputs an image obtained by combining the captured image resized based on the input format of the diagnostic model and the cropped image into the diagnostic model, and performs the diagnosis based on the diagnostic result output by the diagnostic model. [Appendix 6] 6. The image processing device according to claim 1, wherein the diagnostic means performs a diagnosis regarding at least one of a serrated lesion, a hyperplastic polyp, an adenoma, and an intraepithelial carcinoma. [Appendix 7] 7. The image processing device according to claim 1, wherein the examination target is a large intestine or a stomach. [Appendix 8] 8. The image processing device according to any one of claims 1 to 7, further comprising an output control means for causing a display device or a sound output device to output information based on the diagnosis. [Appendix 9] The image processing device described in Appendix 8, wherein, when the output control means detects the presence of a specific type of lesion through the diagnosis, the output control means causes the display device or the sound output device to output at least information notifying of the presence of the lesion as information based on the diagnosis. [Appendix 10] The image processing device according to claim 8 or 9, wherein the output control means causes the display device or the sound output device to output information regarding at least one of the classification or risk of the lesion site based on the diagnosis as information based on the diagnosis. [Appendix 11] The computer An image of the object to be examined is obtained by an imaging unit provided in the endoscope; Detecting a lesion site suspected of being a lesion in the captured image; performing a diagnosis on the lesion site based on the photographed image and a cut-out image obtained by cutting out the lesion site from the photographed image; Image processing methods. [Appendix 12] An image of the object to be examined is obtained by an imaging unit provided in the endoscope; Detecting a lesion site suspected of being a lesion in the captured image; A storage medium storing a program for causing a computer to execute a process for diagnosing the lesion site based on the photographed image and a cropped image of the lesion site cropped from the photographed image.

[0072] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0073] 1. 1X image processing device 2 Display device 3 Endoscope 4. Server equipment 11 Processors 12. Memory 13. Interface 14 Input section 15 Light source section 16 Sound output section 100, 100A Endoscopy System

Claims

1. An acquisition means for acquiring an image of an object to be examined by an imaging unit provided in the endoscope; a lesion site detection means for detecting a lesion site suspected of being a lesion in the photographed image; a diagnostic means for inputting an image obtained by combining the photographed image and a cut-out image of the lesion area from the photographed image into a diagnostic model that outputs a diagnostic result regarding a lesion area in the image when the photographed image is input, and acquiring the diagnostic result output by the diagnostic model; an output control means for outputting information based on the diagnosis result to a display device or a sound output device; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the diagnostic means inputs an image in which the photographed image and the cropped image are superimposed in the channel direction to the diagnostic model, and acquires the diagnostic result output by the diagnostic model.

3. The image processing device according to claim 1 , wherein the diagnostic means inputs an image in which the captured image and the cropped image are combined vertically or horizontally into the diagnostic model, and acquires the diagnostic result output by the diagnostic model.

4. The image processing device according to any one of claims 1 to 3, wherein the diagnostic means inputs an image obtained by combining the captured image resized based on the input format of the diagnostic model and the cropped image into the diagnostic model, and obtains the diagnostic result output by the diagnostic model.

5. 5. The image processing device according to claim 1, wherein the diagnostic means obtains the diagnostic result relating to at least one of a serrated lesion, a hyperplastic polyp, an adenoma, and an intraepithelial carcinoma.

6. The image processing device according to any one of claims 1 to 5, wherein the object to be examined is the large intestine or a stomach.

7. the diagnosis result is classification information indicating a classification of the lesion, The information based on the diagnosis result is The classification information, or Information on the name of the lesion or the risk of the lesion based on the classification information; The image processing device according to any one of claims 1 to 6, further comprising at least one of the following:

8. The image processing device according to any one of claims 1 to 7, wherein the output control means, when detecting the presence of a specific type of lesion based on the diagnosis result, causes the display device or the sound output device to output at least information notifying of the presence of the lesion as information based on the diagnosis result.

9. The computer An image of the object to be examined is obtained by an imaging unit provided in the endoscope; Detecting a lesion site suspected of being a lesion in the captured image; an image obtained by combining the photographed image and a cut-out image of the lesion area cut out from the photographed image is input to a diagnostic model that outputs a diagnostic result regarding a lesion area in the image when the image is input, and the diagnostic result output by the diagnostic model is obtained; outputting information based on the diagnosis result to a display device or a sound output device; Image processing methods.

10. An image of the object to be examined is obtained by an imaging unit provided in the endoscope; Detecting a lesion site suspected of being a lesion in the captured image; an image obtained by combining the photographed image and a cut-out image of the lesion area cut out from the photographed image is input to a diagnostic model that outputs a diagnostic result regarding a lesion area in the image when the image is input, and the diagnostic result output by the diagnostic model is obtained; A program that causes a computer to execute a process of outputting information based on the diagnosis result to a display device or a sound output device.

Images