Image processing device, image processing method and program

Abstract

An image processing device 1X comprises a classification means 31X, an image selection means 33X, and a region extraction means 34X. The classification means 31X classifies each of a time series of captured images, which have been obtained by imaging a test object with an imaging unit that is provided on an endoscope, according to whether or not a site of interest is included. The image selection means 33X selects a target image, from which a region of the site of interest is to be extracted, from among the captured images on the basis of the classification result. The region extraction means 34X extracts the region of the site of interest from the target image.

Description

[Technical field]

[0001] The present disclosure relates to the technical field of an image processing device, an image processing method, and a storage medium that process images acquired in an endoscopic examination. [Background technology]

[0002] Conventionally, an endoscope system that displays an image of the inside of an organ lumen has been known. For example, Patent Document 1 discloses a learning method of a learning model that outputs information about a lesion site included in captured image data when the captured image data generated by an imaging device is input. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication WO2020 / 003607 Summary of the Invention [Problem to be solved by the invention]

[0004] When using a trained model to detect the area of ​​a lesion from images taken during an endoscopic examination, the processing time required to extract the area of ​​the lesion is longer than the interval between image captures, and the more accurate the model, the more difficult real-time processing becomes.

[0005] In consideration of the above-mentioned problems, an object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium capable of extracting an area of ​​interest, such as a lesion site, from an image captured during an endoscopic examination. [Means for solving the problem]

[0006] One aspect of the image processing device is When an image is input, whether or not there is a noteworthy part in the image Binary classification results showing It was machine-learned to output Two Valuesa classification means for classifying, based on the classification model, each of time-series photographed images of an object to be examined, obtained by an imaging unit provided in an endoscope, as to whether or not the image includes the site of interest; an image selection means for selecting, from the photographed images, a target image from which the region of interest is to be extracted based on a result of the classification; Based on a segmentation model, which is a model that extracts an area of ​​interest in an image when the image is input and outputs the extraction result of the area, an area extraction means for extracting an area of ​​the portion of interest from the target image; The image processing device has the following features.

[0007] One aspect of the image processing method includes: When an image is input, whether or not there is a noteworthy part in the image Binary classification results showing It was machine-learned to output Two Values Based on the classification model, each of the images in a time series captured by an imaging unit of the endoscope of the object to be examined is classified as to whether or not the image includes the site of interest; selecting a target image from which the region of interest is to be extracted from the captured images based on a result of the classification; Based on a segmentation model, which is a model that extracts an area of ​​interest in an image when the image is input and outputs the extraction result of the area, Extracting the region of interest from the target image; Image processing methods.

[0008] One aspect of the program is When an image is input, whether or not there is a noteworthy part in the image Binary classification results showing It was machine-learned to output Two Values Based on the classification model, each of the images in a time series captured by an imaging unit of the endoscope of the object to be examined is classified as to whether or not the image includes the site of interest; selecting a target image from which the region of interest is to be extracted from the captured images based on a result of the classification; Based on a segmentation model, which is a model that extracts an area of ​​interest in an image when the image is input and outputs the extraction result of the area, The program causes a computer to execute a process of extracting the region of the site of interest from the target image. Effect of the Invention

[0009] According to the present disclosure, it is possible to suitably extract a region of interest from an endoscopic image. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a schematic configuration of an endoscopic examination system. [Diagram 2] 2 shows the hardware configuration of an image processing device. [Diagram 3] (A) Schematic diagram of a binary classification model. (B) Schematic diagram of a segmentation model. [Figure 4] FIG. 2 is a functional block diagram of the image processing device. [Diagram 5] FIG. 1 is a schematic diagram of a method for selecting an image to be segmented based on confidence levels. [Figure 6] FIG. 1 is a schematic diagram of a method for selecting an image to be segmented based on a feature vector. [Figure 7] 4 shows an example of a display screen displayed by a display device during an endoscopic examination. [Figure 8] 5 is an example of a flowchart showing an overview of a display process executed by the image processing device during an endoscopic examination in the first embodiment. [Figure 9] FIG. 13 is a schematic configuration diagram of an endoscopic inspection system according to a modified example. [Figure 10] FIG. 11 is a block diagram of an image processing device according to a second embodiment. [Figure 11] 13 is an example of a flowchart executed by the image processing device in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. As shown in Fig. 1, the endoscopic examination system 100 is a system that presents a site suspected of being a lesion (lesion site) to an examiner such as a doctor who performs examination or treatment using an endoscope, and mainly includes an image processing device 1, a display device 2, and an endoscope scope 3 connected to the image processing device 1.

[0013] The image processing device 1 acquires images (also referred to as "captured images Ia") captured by the endoscope 3 in time series from the endoscope 3, and displays a screen based on the captured images Ia on the display device 2. The captured images Ia are images captured at a predetermined time interval during at least one of the steps of inserting or ejecting the endoscope 3 into the subject. In this embodiment, the image processing device 1 analyzes the captured images Ia to extract an area in the captured images Ia that corresponds to a lesion site (also referred to as a "lesion image area") from the captured images Ia, and displays information related to the extracted lesion image area on the display device 2.

[0014] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0015] The endoscope scope 3 mainly comprises an operation unit 36 ​​through which the examiner makes predetermined inputs, a flexible shaft 37 that is inserted into the subject's organ to be imaged, a tip portion 38 that incorporates an imaging unit such as an ultra-miniature imaging element, and a connection portion 39 for connecting to the image processing device 1.

[0016] In the following, a process in an endoscopic examination of the large intestine will be described as a representative example, but the examination target is not limited to the large intestine, and may be the esophagus or stomach. Examples of endoscopes that are targets of the present disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, angioscopes, and epidural endoscopes. Examples of pathological conditions of lesion sites that are targets of detection in the present disclosure include the following (a) to (f). (a) Head and neck: pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: Esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor (c) Stomach: stomach cancer, gastritis, stomach ulcer, stomach polyps, stomach tumors (d) Duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease (f) Colon: colon cancer, colon neoplastic disease, colon inflammatory disease, colon polyps, colon polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0017] (2) Hardware Configuration 2 shows a hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0018] The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0019] The memory 12 is composed of various volatile memories used as working memories, such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and a non-volatile memory that stores information necessary for processing of the image processing device 1. The memory 12 may include an external storage device, such as a hard disk connected to or built into the image processing device 1, or may include a storage medium, such as a removable flash memory. The memory 12 stores a program for the image processing device 1 to execute each process in this embodiment. The memory 12 also stores binary classification model information D1 and segmentation model information D2. Details of these data will be described later. At least one of the binary classification model information D1 and the segmentation model information D2 may be stored in an external device capable of data communication with the image processing device 1 via wired or wireless communication, instead of the memory 12.

[0020] The interface 13 performs an interface operation between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the captured image Ia supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for communicating with an external device by wire or wirelessly, or may be a hardware interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0021] The input unit 14 generates an input signal based on an operation by an examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, a voice input device, etc. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also include a built-in pump for sending out water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0022] (3) Data Overview Next, the binary classification model information D1 and the segmentation model information D2 will be described.

[0023] The binary classification model information D1 is information on a binary classification model that outputs information on classification of whether or not the photographed image Ia has a lesion site. The binary classification model information D1 includes, for example, parameters required to configure a binary classification model. The binary classification model is an arbitrary machine learning model or statistical model, and is configured to output information (classification information) on whether or not a lesion site exists in the input photographed image Ia when the photographed image Ia is input. The binary classification model may output two values ​​(for example, a value of 0 or 1) according to whether or not a lesion site exists, or may output a set of a certainty that a lesion site exists and a certainty that a lesion site does not exist, or may output both of these. The "certainty that a lesion site exists" and the "certainty that a lesion site does not exist" correspond to the "certainty regarding the presence or absence of a lesion site" described later. In addition, the "certainty that a lesion site exists" is a value indicating the degree of possibility that a lesion site exists in the photographed image Ia, and the higher the possibility, the higher the degree of possibility. The "certainty that a lesion site does not exist" is a value indicating the degree of possibility that a lesion site does not exist in the photographed image Ia, and the higher the degree of possibility, ... A binary classification model is an example of a "classification model".

[0024] Fig. 3(A) is a schematic diagram of a binary classification model configured based on binary classification model information D1. In Fig. 3(A), the binary classification model has a feature extractor and a score calculator. When a captured image Ia is input, the feature extractor extracts features of a predetermined number of dimensions and supplies a feature vector representing the extracted features to the score calculator. The score calculator outputs a score (in Fig. 3(A), the certainty that there is a lesion area and the certainty that there is no lesion area) in response to the feature vector input from the feature extractor.

[0025] The binary classification model may be, for example, a model based on logistic regression, k-nearest neighbors, boosting, decision tree, neural network, support vector machine, etc. Furthermore, as the architecture of the neural network, for example, AlexNet, VGG, ResNet, SqueezeNet, DenseNet, Inception, GoogLeNet, ShuffleNet, MobileNet, ResNeXt, Wide ReNet, NASNet, etc. may be adopted. When the binary classification model is configured by a neural network, the binary classification model information D1 includes various parameters such as a layer structure, a neuron structure of each layer, the number of filters and filter size in each layer, and the weight of each element of each filter.

[0026] The segmentation model information D2 is information about a segmentation model that extracts an area (also called a "lesion image area Tc") in the captured image Ia that constitutes a lesion site from an input image. The segmentation model information D2 includes parameters necessary for configuring the segmentation model. The segmentation model is, for example, a machine learning model such as a neural network, and is configured to output an extraction result of the lesion image area Tc in the input captured image Ia when the captured image Ia is input. The extraction result of the lesion image area Tc output by the segmentation model may be, for example, a mask image of the lesion image area Tc (a binary image in which pixels corresponding to the lesion image area Tc have different values ​​from other pixels), or may be a reliability map that is a map on the image that indicates the reliability of the lesion image area Tc.

[0027] Fig. 3(B) is a schematic diagram of a segmentation model configured based on the segmentation model information D2. In Fig. 3(B), as an example, when a captured image Ia selected by the image processing device 1 is input, the segmentation model outputs a mask image representing a lesion image region Tc in the input captured image Ia as an extraction result of the lesion image region Tc.

[0028] Representative models of neural networks used in segmentation models include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab.

[0029] (4) Extraction and display of lesion image area Next, a description will be given of the process of extracting and displaying the lesion image region Tc executed by the image processing device 1. In summary, the image processing device 1 applies a segmentation model to the captured image Ia selected based on information output by the binary classification model, and displays the extraction result of the lesion image region Tc by the segmentation model on the display device 2. In this way, the image processing device 1 extracts and displays the lesion image region Tc with high accuracy while suitably reducing the processing load.

[0030] Here, a supplementary explanation will be given on the relationship between the binary classification model and the segmentation model. The binary classification model does not extract the lesion site, so the processing load is lower than that of the segmentation model, and the processing time is shorter than the interval based on the frame rate of the photographed image Ia. Therefore, the binary classification model is suitable for real-time processing. On the other hand, the segmentation model extracts the lesion site (determines the lesion image area Tc), so the processing load is higher than that of the binary classification model, and the processing time tends to be longer than the interval based on the frame rate of the photographed image Ia. This tendency becomes more pronounced as a high-precision segmentation model is used. Therefore, if the segmentation model is applied to all photographed images Ia, the processing load of the image processing device 1 becomes excessive, and real-time processing cannot be performed. In consideration of the above, in this embodiment, the binary classification model is applied to all photographed images Ia, and the segmentation model is selectively applied according to the information output by the binary classification model. As a result, the image processing device 1 extracts and presents the lesion image area Tc with high precision while maintaining real-time processing.

[0031] Fig. 4 is a functional block diagram of the image processing device 1. As shown in Fig. 4, the processor 11 of the image processing device 1 functionally includes a photographed image acquisition section 30, a classification section 31, a lesion candidate section detection section 32, an image selection section 33, a region extraction section 34, and a display control section 35. Note that in Fig. 4, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to that in Fig. 3. The same applies to other functional block diagrams described later.

[0032] The captured image acquisition unit 30 acquires the captured images Ia captured by the endoscope 3 at predetermined intervals via the interface 13. Then, the captured image acquisition unit 30 supplies the acquired captured images Ia to the classification unit 31, the image selection unit 33, and the display control unit 35, respectively.

[0033] The classification unit 31 classifies each captured image Ia acquired by the captured image acquisition unit 30 as to whether or not a lesion site is included, and supplies the classification result "Rc" to the lesion candidate section detection unit 32 and the image selection unit 33. In this case, the classification unit 31 configures a binary classification model by referring to the binary classification model information D1, and obtains the classification result Rc by inputting the captured image Ia acquired by the captured image acquisition unit 30 to the configured binary classification model. Note that the classification unit 31 may include a feature vector output by a feature extractor of the binary classification model in the classification result Rc supplied to the image selection unit 33, in addition to the degree of certainty of the presence or absence of a lesion site. This feature vector is used by the image selection unit 33 to select the captured image Ia, as described later.

[0034] Based on the classification result Rc for each photographed image Ia supplied from the classification unit 31, the lesion candidate section detection unit 32 detects a section (also referred to as a "lesion candidate section St") in which photographed images Ia classified as including a lesion section are consecutive. In this case, for example, when the latest photographed image Ia is classified as an image including a lesion section (including the case where the certainty of the presence of a lesion section is higher than the certainty of the absence of a lesion section), the lesion candidate section detection unit 32 detects the presence of a lesion candidate section St including the photographed image Ia. On the other hand, when the lesion candidate section St is being detected, if the latest photographed image Ia is classified as an image not including a lesion section, the lesion candidate section detection unit 32 determines that the lesion candidate section St has ended. Then, the lesion candidate section detection unit 32 supplies the section detection result "Rd" related to the lesion candidate section St to the image selection unit 33.

[0035] The image selection unit 33 selects an image (also called a "segmentation target image Itag") to which a segmentation model is to be applied from the captured image Ia corresponding to the lesion candidate section St based on the classification result Rc and the section detection result Rd. A method for selecting the segmentation target image Itag will be described later. The image selection unit 33 supplies the selected segmentation target image Itag to the region extraction unit 34.

[0036] The region extraction unit 34 performs a process of extracting the lesion image region Tc from the segmentation target image Itag selected by the image selection unit 33. In this case, the region extraction unit 34 acquires the extraction result of the lesion image region Tc by inputting the segmentation target image Itag to the segmentation model specified based on the segmentation model information D2. Then, the region extraction unit 34 supplies the region extraction result “Re” related to the lesion image region Tc to the display control unit 35.

[0037] The display control unit 35 generates display information Ib based on the photographed image Ia and the region extraction result Re, and supplies the display information Ib to the display device 2 via the interface 13, thereby causing the display device 2 to display the photographed image Ia and information related to the lesion image region Tc. An example of the display by the display control unit 35 will be described later.

[0038] Here, each of the components of the photographed image acquisition unit 30, the classification unit 31, the lesion candidate section detection unit 32, the image selection unit 33, the region extraction unit 34, and the display control unit 35 can be realized, for example, by the processor 11 executing a program. Also, each component may be realized by recording a necessary program in an arbitrary non-volatile storage medium and installing it as necessary. Note that at least a part of these components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Also, at least a part of these components may be realized by using an integrated circuit that is programmable by a user, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program composed of each of the above components. Also, at least a part of each component may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The above is the same in other embodiments described later. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0039] (5) Selecting images for segmentation Next, a method of selecting the segmentation target image Itag executed by the image selection unit 33 when the lesion candidate section detection unit 32 detects the lesion candidate section St will be specifically described. The image selection unit 33 selects the segmentation target image Itag based on the classification result Rc of each captured image Ia in order to select an image of a portion where the output of the binary classification model is stable in the lesion candidate section St. By selecting an image of a portion where the output of the binary classification model is stable as the segmentation target image Itag in this way, it becomes possible to select a segmentation target image Itag with little blurring or shaking of the image caused by, for example, the endoscope operation by the examiner. With this image selection, it is expected that the lesion image region Tc can be extracted with high accuracy by applying the segmentation model. In addition, it becomes possible to present an image suitable as a display image to be presented to the examiner based on the lesion image region Tc extracted with high accuracy in this way.

[0040] In this embodiment, when the condition regarding the classification result Rc of each photographed image Ia is satisfied for a predetermined number of consecutive images in time series, the image selection unit 33 selects the segmentation target image Itag from the predetermined number of photographed images Ia. This makes it possible to select the segmentation target image Itag with less blurring or shaking of the image caused by the endoscope operation by the examiner. In this case, the condition regarding the classification result Rc may be a condition regarding the confidence level regarding the presence or absence of a lesion site in the photographed image Ia, or may be a condition based on a feature vector extracted from the photographed image Ia. In this manner, the image selection unit 33 may select the segmentation target image Itag based on the confidence level regarding the presence or absence of a lesion site in the photographed image Ia, or may select the segmentation target image Itag based on the feature vector extracted from the photographed image Ia. Hereinafter, a method of selecting the segmentation target image Itag based on the confidence level regarding the presence or absence of a lesion site in the photographed image Ia and a method of selecting the segmentation target image Itag based on the feature vector extracted from the photographed image Ia will be described in order.

[0041] (5-1) Selection method based on confidence level The image selection unit 33 may determine whether or not a segmentation model needs to be applied based on the confidence level regarding the presence or absence of a lesion area output by the binary classification model, and select a segmentation target image Itag from the captured images Ia that constitute the lesion candidate section St.

[0042] In this case, the image selection unit 33 determines that the application conditions of the segmentation model are satisfied when a predetermined number "N" (N is an integer equal to or greater than 2) of consecutive captured images Ia in which the certainty of the presence of a lesion is equal to or greater than a predetermined threshold (also referred to as "first threshold t1") and the difference in the certainty of the presence of a lesion is less than a predetermined threshold (also referred to as "second threshold t2"). Then, the image selection unit 33 selects the Nth (i.e., the latest) captured image Ia from among the N captured images Ia that satisfy the application conditions of the segmentation model as the segmentation target image Itag. The first threshold t1, the second threshold t2, and the number N relating to the application conditions of the segmentation model are determined in advance, for example, taking into consideration the classification accuracy of the binary classification model, and are stored in advance in the memory 12, etc.

[0043] Fig. 5 is a schematic diagram of a method for selecting an image Itag to be segmented based on the confidence level. In Fig. 5, the latest captured image Ia acquired at time "T" and six captured images Ia acquired immediately before the captured image Ia are displayed in association with the times of acquisition "T" to "T-6". In Fig. 5, the captured images Ia for which the confidence level of the presence of a lesion is equal to or greater than the first threshold value t1 are indicated by hatching. Here, the number N is set to "3".

[0044] In FIG. 5, the confidence level of the lesion area is equal to or higher than the first threshold value t1 for the captured image Ia acquired at time T, the captured image Ia acquired at time "T-1", and the captured image Ia acquired at time "T-2". The difference between the confidence levels of the lesion area among these captured images Ia (i.e., the difference between the maximum and minimum confidence levels of the lesion area among these captured images Ia) is less than the second threshold value t2. Therefore, since there are a predetermined number N consecutive captured images Ia in which the confidence level of the lesion area is equal to or higher than the first threshold value t1 and the difference between the confidence levels of the lesion area is less than the second threshold value t2, the image selection unit 33 selects the captured image Ia acquired at time T, which is the Nth captured image Ia, as the segmentation target image Itag. This allows the image selection unit 33 to suitably select the captured image Ia in which the output of the binary classification model is stable as the segmentation target image Itag.

[0045] Incidentally, although the certainty of the presence of a lesion area for the photographed image Ia acquired at time "T-5" is equal to or greater than the first threshold value t1, the certainty of the presence of a lesion area for the successive photographed images Ia is less than the first threshold value t1, and does not satisfy the application condition of the segmentation model for the number N. Therefore, the photographed image Ia acquired at time "T-5" is not selected as the segmentation target image Itag.

[0046] Here, a variation of the method for selecting the segmentation target image Itag based on the above-mentioned confidence factor will be described.

[0047] For example, the image selection unit 33 may dynamically change the number N according to the selection of the segmentation target image Itag. For example, after selecting a segmentation target image Itag in a certain lesion candidate section St, the image selection unit 33 sets the number N in the lesion candidate section St to be larger by a predetermined number. Then, when the application condition of the segmentation model after increasing the number N is satisfied, the image selection unit 33 again selects the segmentation target image Itag.

[0048] 5, the image selection unit 33 selects the photographed image Ia at time T as the segmentation target image Itag, and then changes the number N from "3" to "5". Then, when the confidence that the photographed images Ia acquired at time "T+1" and time "T+2" have a lesion area is equal to or greater than a first threshold value t1, and the difference in confidence that the photographed images Ia acquired at time T-2 to time T+2 have a lesion area is less than a second threshold value t2, the image selection unit 33 selects the photographed image Ia acquired at time "T+2" as the segmentation target image Itag. Then, the segmentation target image Itag selected by the image selection unit 33 is supplied to the region extraction unit 34.

[0049] According to this example, the image selection unit 33 additionally selects the segmentation target image Itag in the lesion candidate section St, and can suitably update the display of the lesion image region Tc to be displayed on the display device 2. On the other hand, when the image selection unit 33 recognizes, based on the section detection result Rd generated by the lesion candidate section detection unit 32, that the latest captured image Ia does not belong to the lesion candidate section St, it resets the number N to the initial value "3".

[0050] In addition, when increasing the number N, the image selection unit 33 may change at least one of the first threshold t1 or the second threshold t2. For example, when increasing the number N, the image selection unit 33 increases the first threshold t1 by a predetermined value or decreases the second threshold t2 by a predetermined value. This allows adjustment so that segmentation target images Itag representing the same lesion site are less likely to be selected (i.e., display update of the lesion image region Tc is less likely). On the other hand, when increasing the number N, the image selection unit 33 decreases the first threshold t1 by a predetermined value or increases the second threshold t2 by a predetermined value. This allows adjustment so that segmentation target images Itag representing the same lesion site are more likely to be selected (i.e., display update of the lesion image region Tc is more likely).

[0051] In another example of changing the number N, the image selection unit 33 may set the number N to a value smaller than the initial value when a lesion candidate section St is detected again immediately after the end of the lesion candidate section St. For example, in the example of Fig. 5, when the lesion candidate section St ends because the confidence level of the lesion section presence in the photographed image Ia acquired at time T+1 becomes lower than the confidence level of the lesion section presence, and the lesion candidate section St is detected again within a predetermined number of frames immediately thereafter, the image selection unit 33 sets the number N to a predetermined value smaller than the initial value. This allows the image selection unit 33 to select the segmentation target image Itag early by reducing the number N, even if the confidence level of the lesion section presence becomes temporarily lower due to image flicker or the like.

[0052] Furthermore, when the application conditions of the segmentation model are satisfied, the image selection unit 33 may select the segmentation target image Itag from the captured images Ia other than the Nth captured image Ia. For example, the image selection unit 33 may select the captured image Ia with the highest degree of certainty that a lesion exists as the segmentation target image Itag from among the N captured images Ia that satisfy the application conditions of the segmentation model.

[0053] (5-2) Selection method based on feature vector The image selection unit 33 may determine whether or not a segmentation model needs to be applied based on the feature vector output by the feature extractor of the binary classification model, and select a segmentation target image Itag from the captured images Ia that constitute the lesion candidate section St.

[0054] For example, the image selection unit 33 calculates the similarity (e.g., cosine similarity) between two consecutive captured images Ia based on the inner product of the feature vectors of these images. The image selection unit 33 then calculates the similarity for each of N consecutive captured images Ia, and if all of these similarities are equal to or greater than a predetermined threshold (also referred to as the "third threshold t3"), it determines that the application conditions for the segmentation model are satisfied. In this case, the image selection unit 33 selects the Nth (i.e., the latest) captured image Ia as the segmentation target image Itag.

[0055] Fig. 6 is a schematic diagram of a method for selecting an image Itag to be segmented based on a feature vector. In Fig. 6, similar to Fig. 5, the latest captured image Ia captured at time T and six captured images Ia captured at times T-1 to T-6 immediately before the captured image Ia are displayed in association with the capture times. Here, the number N is set to "3".

[0056] In FIG. 6, the similarity based on the inner product of the feature vectors of two consecutive captured images Ia is calculated. Then, the image selection unit 33 compares the calculated similarity with a third threshold t3 and determines whether the similarity equal to or greater than the third threshold t3 is satisfied for N consecutive images (three images in this case). Then, the image selection unit 33 determines that the application condition of the segmentation model is satisfied for the N consecutive captured images Ia from time T-2 to time T, since the similarity between the captured images Ia acquired at time T-2 and time T-1 and the similarity between the captured images Ia acquired at time T-1 and time T are both equal to or greater than the third threshold t3. Therefore, in this case, the image selection unit 33 selects the captured image Ia acquired at time T as the segmentation target image Itag.

[0057] Even in this case, the image selection unit 33 can suitably select the captured image Ia in a portion where the output of the binary classification model is stable as the segmentation target image Itag. Note that, similar to the method of selecting the segmentation target image Itag based on the confidence level, the image selection unit 33 may dynamically change the number N. Furthermore, when increasing the number N in response to the selection of the segmentation target image Itag, the image selection unit 33 may adjust the update frequency of the lesion image region Tc by similarly changing the third threshold value t3. Furthermore, instead of selecting the Nth captured image Ia as the segmentation target image Itag, the image selection unit 33 may select the segmentation target image Itag from the N captured images Ia other than the captured image Ia based on the confidence level.

[0058] The image selection unit 33 may combine the method of selecting the segmentation target image Itag based on the confidence level and the method of selecting the segmentation target image Itag based on the feature vector. In this case, the image selection unit 33 selects the segmentation target image Itag when both the application conditions of the segmentation model based on the confidence level and the application conditions of the segmentation model based on the feature vector are satisfied.

[0059] (6) Display Control Next, the display control of the display device 2 executed by the display control unit 35 will be described.

[0060] Fig. 7 shows an example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 35 of the image processing device 1 transmits display information Ib generated based on the captured image Ia acquired by the captured image acquisition unit 30 and the area extraction result Re generated by the area extraction unit 34 to the display device 2, thereby causing the display device 2 to display the display screen shown in Fig. 7.

[0061] In the example of FIG. 7, the display control unit 35 of the image processing device 1 provides on the display screen a latest image display area 70 showing the latest photographed image Ia acquired by the photographed image acquisition unit 30, and a lesion site display area 71 showing a mask image showing a lesion image area Tc based on the area extraction result Re. In this case, the lesion candidate section detection unit 32 detects that it is a lesion candidate section St, and the image selection unit 33 judges the application condition of the segmentation model based on the confidence factor or the feature vector, and selects the segmentation target image Itag. Then, the area extraction unit 34 applies the segmentation model to the selected segmentation target image Itag to extract the lesion image area Tc, and the display control unit 35 displays the mask image showing the lesion image area Tc extracted by the area extraction unit 34 in the lesion site display area 71. The display control unit 35 also displays a video based on the latest photographed image Ia acquired by the photographed image acquisition unit 30 on the latest image display area 70.

[0062] In this way, the display controller 35 can appropriately assist the examiner in identifying the lesion site by presenting the corresponding lesion image region Tc together with the latest captured image Ia.

[0063] In the display example of Fig. 7, the display control unit 35 may highlight the detected lesion image region Tc on the latest captured image Ia instead of providing the latest image display region 70 and the lesion site display region 71, or after providing the latest image display region 70 and the lesion site display region 71. In this case, the display control unit 35 may highlight the lesion image region Tc on the captured image Ia to be displayed, for example, by applying a border effect to the edge portion of the lesion image region Tc on the basis of a mask image representing the lesion image region Tc, or may highlight the entire portion corresponding to the lesion image region Tc by setting a predetermined pixel value. Moreover, instead of displaying a mask image of the lesion image region Tc in the lesion site display region 71, the display control unit 35 may display a reliability map showing the reliability of whether or not each pixel corresponds to the lesion image region Tc.

[0064] Next, the update timing of the image displayed in the lesion site display area 71 will be described.

[0065] Let us consider a case where, after an image representing a lesion image region Tc is displayed in lesion site display region 71, image selection unit 33 selects a new segmentation target image Itag, and region extraction unit 34 generates region extraction result Re, while the lesion candidate section St is continuing. In this case, display control unit 35 updates the display of lesion site display region 71 with an image representing lesion image region Tc based on the newly generated region extraction result Re. This allows display control unit 35 to suitably update the display of lesion site display region 71 based on the latest segmentation target image Itag selected by image selection unit 33.

[0066] Here, when the image selection unit 33 newly selects a segmentation target image Itag and the region extraction unit 34 generates a region extraction result Re, the display control unit 35 may determine whether or not it is necessary to update the display of the lesion site display area 71 based on the area of ​​the lesion image area Tc. Specifically, when the area of ​​the lesion image area Tc represented by the newly generated region extraction result Re is larger than the area of ​​the lesion image area Tc being displayed in the lesion site display area 71 (i.e., the number of pixels forming the lesion image area Tc), the display control unit 35 updates the display of the lesion site display area 71 based on the newly generated region extraction result Re. In this way, when a plurality of segmentation target images Itag are selected in a series of lesion candidate sections St, the display control unit 35 determines the segmentation target image Itag for displaying the lesion image area Tc based on the area of ​​the corresponding lesion image area Tc. In this way, the display control unit 35 can display an image that preferably represents the lesion site in the lesion site display area 71.

[0067] Furthermore, when the latest captured image Ia no longer belongs to the lesion candidate section St, the display control unit 35 may stop displaying the image in the lesion section display area 71. In this case, when the display control unit 35 detects that the lesion candidate section St has ended based on the section detection result Rd generated by the lesion candidate section detection unit 32, the display control unit 35 hides the image being displayed in the lesion section display area 71. In this way, the display control unit 35 suitably prevents the continuous display of an image clearly showing a lesion section that is no longer being photographed by the endoscope scope 3.

[0068] Furthermore, when display control unit 35 detects a predetermined input based on the operation of input unit 14 by the examiner, it may prevent the display of the image in lesion site display area 71 from being updated (i.e., stop the update). In this case, display control unit 35 continues (fixes) the display of the image displayed in lesion site display area 71 at the time when the display control unit 35 detects the predetermined input based on the operation of input unit 14 by the examiner. Then, when display control unit 35 detects a predetermined input that permits the update of the display of the image in lesion site display area 71, it resumes the update of the display of the image in lesion site display area 71. In this way, by stopping the update of the display regarding lesion image area Tc due to an external input based on the operation of input unit 14 by the examiner, display control unit 35 can give the examiner sufficient time to check the display regarding lesion image area Tc.

[0069] (7) Processing Flow FIG. 8 is an example of a flowchart showing an overview of the processing executed by the image processing device 1 during endoscopic examination in the first embodiment.

[0070] First, the photographed image acquisition section 30 of the image processing device 1 acquires the photographed image Ia (step S11). In this case, the photographed image acquisition section 30 of the image processing device 1 receives the photographed image Ia from the endoscope scope 3 via the interface 13.

[0071] Next, the classification unit 31 of the image processing device 1 classifies the photographed image Ia acquired in step S11 by the binary classification model configured based on the binary classification model information D1 (step S12). The lesion candidate section detection unit 32 of the image processing device 1 determines whether or not the photographed image Ia corresponds to the lesion candidate section St based on the classification result Rc generated by the classification unit 31 (step S13). If the photographed image corresponds to the lesion candidate section St (step S13; Yes), the process proceeds to step S14. On the other hand, if the photographed image Ia does not correspond to the lesion candidate section St (step S13; No), the display control unit 35 causes the display device 2 to display the photographed image Ia acquired in step S11 (step S18). Note that, when the display control unit 35 is executing display of the lesion image region Tc based on step S17 described later, it is preferable that the display control unit 35 erases the display of the lesion image region Tc.

[0072] On the other hand, if the section corresponds to the lesion candidate section St, the image selection unit 33 of the image processing device 1 determines whether or not the application condition of the segmentation model is satisfied (step S14). In this case, for example, the image selection unit 33 selects “(5) Selecting images for segmentation In step S14, the image selection unit 33 determines whether the application condition of the segmentation model based on at least one of the confidence level and the feature vector described in the section "is satisfied or not. If it is determined that the application condition of the segmentation model is satisfied (step S14; Yes), the image selection unit 33 selects the segmentation target image Itag (step S15). In this case, the image selection unit 33 may select the latest captured image Ia as the segmentation target image Itag, or may select the segmentation target image Itag based on the confidence level of the presence of a lesion site. Then, the region extraction unit 34 applies the segmentation model configured based on the segmentation model information D2 to the segmentation target image Itag, thereby extracting the lesion image region Tc (step S16). Then, the display control unit 35 causes the display device 2 to display the latest captured image Ia and the lesion image region Tc extracted in step S16 (step S17).

[0073] After step S17 or step S18, the image processing device 1 determines whether or not the endoscopic examination has ended (step S19). For example, when the image processing device 1 detects a predetermined input to the input unit 14 or the operation unit 36, the image processing device 1 determines that the endoscopic examination has ended. Then, when the image processing device 1 determines that the endoscopic examination has ended (step S19; Yes), the image processing device 1 ends the processing of the flowchart. On the other hand, when the image processing device 1 determines that the endoscopic examination has not ended (step S19; No), the image processing device 1 returns the processing to step S11. Then, the image processing device 1 executes the processing of steps S11 to S19 on the captured image Ia newly generated by the endoscope 3.

[0074] (8) Variations Next, preferred modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

[0075] (Variation 1) The image processing device 1 may process a video image constructed from the captured images Ia generated during an endoscopic examination after the examination.

[0076] For example, at any timing after an examination, when an image to be processed is designated based on user input via the input unit 14, the image processing device 1 sequentially performs the process of the flowchart in Fig. 8 on the time-series captured images Ia constituting the designated image. Then, when the image processing device 1 determines in step S19 that the target image has ended, it ends the process of the flowchart, and when the target image has not ended, it returns to step S11 and performs the process of the flowchart on the next captured image Ia in the time series.

[0077] (Variation 2) The binary classification model information D1 and the segmentation model information D2 may be stored in a storage device separate from the image processing device 1.

[0078] Fig. 9 is a schematic configuration diagram of an endoscopic examination system 100A in Modification 2. For simplicity, Fig. 9 does not show the display device 2 and the endoscope 3. The endoscopic examination system 100A shown in Fig. 9 includes a server device 4 that stores binary classification model information D1 and segmentation model information D2. The endoscopic examination system 100A also includes a plurality of image processing devices 1 (1A, 1B, ...) capable of data communication with the server device 4 via a network.

[0079] In this case, each image processing device 1 refers to the binary classification model information D1 and the segmentation model information D2 via the network. In this case, the interface 13 of each image processing device 1 includes a communication interface such as a network adapter for performing communication. In this configuration, each image processing device 1 can refer to the binary classification model information D1 and the segmentation model information D2, as in the above-mentioned embodiment, and suitably execute the extraction process and display process of the lesion image region Tc.

[0080] (Variation 3) The object to be detected by the detection model is not limited to a lesion site, but may be any desired site of interest that the examiner needs to pay attention to. Such a site of interest is not limited to a lesion site, but may be an inflamed site, a site with a surgical scar or other incision, a site with a fold or protrusion, a site where the tip 38 of the endoscope 3 is likely to come into contact (be easily pinched) with the wall surface inside the lumen, and the like.

[0081] (Variation 4) The image processing device 1 may use a classification model that performs three or more classifications instead of a binary classification model. For example, the classification model used by the image processing device 1 may be a model that classifies X+1 types, including "lesion type 1" to "lesion type X" ("X" is an integer equal to or greater than 2) and "non-lesion." In this case, instead of the binary classification model information D1, the memory 12 stores classification model information that is information on a classification model that performs three or more classifications, and the image processing device 1 generates a classification result for the captured image Ia by referring to the classification model information.

[0082] In this way, the classification model used by the image processing device is not limited to a binary classification model as long as it is a model that determines the "presence or absence of a lesion" as its function.

[0083] <Second embodiment> 10 is a block diagram of an image processing device 1X in the second embodiment. The image processing device 1X includes a classification unit 31X, an image selection unit 33X, and a region extraction unit 34X.

[0084] The classification means 31X classifies each of the time-series photographed images of the inspection object captured by the imaging unit provided in the endoscope into whether or not it contains a site of interest that should be noted. The classification means 31X can be the classification unit 31 in the first embodiment (including modified examples, the same applies below). Note that the classification means 31X may instantly acquire the photographed images generated by the imaging unit, or may acquire photographed images generated in advance by the imaging unit and stored in a storage device at a predetermined timing.

[0085] The image selection means 33X selects a target image from which a region of interest is to be extracted from among the captured images based on the result of the classification. The image selection means 33X can be the image selection unit 33 in the first embodiment.

[0086] The region extraction means 34X extracts a region of a site of interest from a target image. The region extraction means 34X can be the region extraction unit 34 in the first embodiment.

[0087] 11 is an example of a flowchart showing a processing procedure in the second embodiment. First, the classification means 31X classifies each of the time-series captured images of the inspection object captured by the imaging unit provided in the endoscope as to whether or not the captured images contain a site of interest that should be noted (step S21). Next, the image selection means 33X selects a target image from which a site of interest area is to be extracted from the captured images based on the classification result (step S22). Then, the area extraction means 34X extracts the site of interest area from the target image (step S23).

[0088] According to the second embodiment, the image processing device 1X limits the target images from which the area of ​​the part of interest is extracted based on the classification result of whether or not the part of interest is included, and can preferably extract the area of ​​the part of interest while reducing the processing load.

[0089] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following notes.

[0090] [Appendix 1] A classification means for classifying each of the time-series photographed images of the object of examination captured by an imaging unit provided in the endoscope as to whether or not it includes a site of interest that should be noted; an image selection means for selecting, from the photographed images, a target image from which the region of interest is to be extracted based on a result of the classification; an area extraction means for extracting an area of ​​the portion of interest from the target image; An image processing device comprising: [Appendix 2] 2. The image processing device according to claim 1, wherein the image selection means selects the target image from the predetermined number of captured images when a condition related to the result of the classification is satisfied for a predetermined number of consecutive images in the time series. [Appendix 3] The image processing device according to claim 1 or 2, wherein the classification means performs the classification for each of the captured images in the time series based on a classification model that outputs information regarding the presence or absence of a site of interest in an image when the image is input. [Appendix 4] The classification model outputs a confidence level regarding the presence or absence of the site of interest; The image processing device according to claim 3, wherein the image selection means selects the target image based on the degree of certainty. [Appendix 5] 5. The image processing device according to claim 4, wherein the image selection means selects the target image from the predetermined number of captured images when a condition regarding the certainty level is satisfied for a predetermined number of consecutive images in the time series. [Appendix 6] The classification means extracts features from the input image, 6. The image processing device according to claim 1, wherein the image selection means selects the target image based on a feature vector obtained by the feature extraction. [Appendix 7] The image processing device according to claim 6, wherein the image selection means selects the target image from a predetermined number of consecutive captured images when a condition regarding the similarity of the captured images calculated based on the feature vector is satisfied for a predetermined number of consecutive captured images in the time series. [Appendix 8] The image processing device according to any one of appendices 2, 5, and 7, wherein the image selection means changes the specified number after selecting the target image in a section in which the captured images classified as including the area of ​​interest are consecutive. [Appendix 9] The image processing device according to any one of claims 1 to 8, wherein the area extraction means extracts the area of ​​the area of ​​interest from the target image based on a segmentation model, which is a model that outputs information regarding the area of ​​the area of ​​interest in an image when the image is input. [Appendix 10] 10. The image processing device according to any one of claims 1 to 9, further comprising a display control means for causing a display device to display information relating to the region of interest. [Appendix 11] The image processing device described in Appendix 10, wherein when multiple target images are selected in a section where the captured images classified as including the area of ​​the interest are consecutive, the display control means determines the target image as the target for displaying information regarding the area of ​​the interest based on the area of ​​the interest. [Appendix 12] 12. The image processing device according to claim 10, wherein the display control means stops updating of the display of information relating to the region of interest based on an external input. [Appendix 13] The computer Classifying each of the images of the object of examination in a time series obtained by photographing the object of examination using an imaging unit provided in the endoscope as to whether or not the image includes a site of interest that should be noted; selecting a target image from which the region of interest is to be extracted from the captured images based on a result of the classification; Extracting the region of interest from the target image; Image processing methods. [Appendix 14] Classifying each of the images of the object of examination in a time series obtained by photographing the object of examination using an imaging unit provided in the endoscope as to whether or not the image includes a site of interest that should be noted; selecting a target image from which the region of interest is to be extracted from the captured images based on a result of the classification; A storage medium storing a program for causing a computer to execute a process for extracting the region of interest from the target image.

[0091] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and corrections that a person skilled in the art could make in accordance with the entire disclosure, including the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated herein by reference. [Explanation of symbols]

[0092] 1. 1X image processing device 2 Display device 3 Endoscope 4. Server equipment 11 Processors 12. Memory 13. Interface 14 Input section 15 Light source section 16 Sound output section 100, 100A Endoscopy System

Claims

1. a classification means for classifying each of a time series of captured images of an object of inspection captured by an imaging unit provided in an endoscope as to whether or not the image contains a site of interest based on a binary classification model that has been machine-learned to output a binary classification result indicating whether or not a site of interest that should be noted is present in the image when the image is input; an image selection means for selecting, from the photographed images, a target image from which the region of interest is to be extracted based on a result of the classification; a region extraction means for extracting a region of a region of interest from a target image based on a segmentation model which is a model for extracting a region of the region of interest from an input image and outputting an extraction result of the region; An image processing device comprising:

2. The image processing device according to claim 1 , wherein the image selection means selects the target image from the predetermined number of captured images when a condition related to the result of the classification is satisfied for a predetermined number of consecutive images in the time series.

3. The binary classification model outputs a confidence level regarding the presence or absence of the site of interest; The image processing device according to claim 1 , wherein the image selection means selects the target image based on the degree of certainty.

4. The image processing device according to claim 3 , wherein the image selection means selects the target image from the predetermined number of captured images when the condition relating to the certainty factor is satisfied for a predetermined number of consecutive images in the time series.

5. The classification means extracts features from the input image, 5. The image processing device according to claim 1, wherein the image selection means selects the target image based on a feature vector obtained by the feature extraction.

6. 6. The image processing device according to claim 5, wherein the image selection means selects the target image from a predetermined number of consecutive captured images when a condition regarding the similarity of the captured images calculated based on the feature vector is satisfied for the predetermined number of consecutive captured images in the time series.

7. The image processing device according to claim 2 , wherein the image selection means changes the predetermined number after selecting the target image in a section in which the photographed images classified as including the site of interest continue.

8. 8. The image processing device according to claim 1, further comprising a display control means for causing a display device to display information relating to the region of interest.

9. 9. The image processing device of claim 8, wherein when multiple target images are selected in a section in which the captured images classified as including the area of ​​interest are consecutive, the display control means determines the target image for displaying information regarding the area of ​​...

10. The image processing device according to claim 8 , wherein the display control means stops updating of the display of the information relating to the region of interest based on an external input.

11. The computer Based on a binary classification model that has been machine-learned to output a binary classification result indicating whether or not a site of interest that should be noted is present in an input image, classifying each of a time series of captured images of an object of examination captured by an imaging unit provided in an endoscope as to whether or not the site of interest is present, selecting a target image from which the region of interest is to be extracted from the captured images based on a result of the classification; Extracting a region of a site of interest from a target image based on a segmentation model, which is a model that extracts a region of a site of interest in an image when the image is input and outputs an extraction result of the region; Image processing methods.

12. Based on a binary classification model that has been machine-learned to output a binary classification result indicating whether or not a site of interest that should be noted is present in an input image, classifying each of a time series of captured images of an object of examination captured by an imaging unit provided in an endoscope as to whether or not the site of interest is present, selecting a target image from which the region of interest is to be extracted from the captured images based on a result of the classification; A program that causes a computer to execute a process of extracting an area of ​​interest from a target image based on a segmentation model, which is a model that extracts an area of ​​interest within an image when the image is input and outputs the extraction result of the area.

Images