Image processing device, image processing method, and program

Abstract

An image processing device 1X includes an acquisition means 30X, a variation detection means 311X, a selection means 312X, and a lesion detection means 34X. The acquisition means 30X acquires an endoscopic image obtained by image-taking of a subject by an imaging unit provided to an endoscope. The variation detection means 311X detects a degree of variation in the endoscopic image. On the basis of the degree of variation, the selection means 312X selects one of a first model that performs inference relating to a lesion of the subject on the basis of a predetermined number of endoscopic images, and a second model that performs inference relating to the lesion on the basis of a variable number of endoscopic images. The lesion detection means 312X detects the lesion on the basis of a selected model that is the first model or the second model that has been selected.

Description

Technical Field

[0001] The present disclosure relates to the technical field of an image processing apparatus, an image processing method, and a storage medium for processing images acquired in an endoscopic examination.

Background Art

[0002] Conventionally, an endoscopic system for displaying an image obtained by photographing the inside of a lumen of an organ is known. For example, Patent Document 1 discloses a method of training a learning model that outputs information regarding a lesion site included in endoscopic image data when the endoscopic image data generated by a photographing device is input. Further, Patent Document 2 discloses a classification method for classifying series data by a method applying a sequential probability ratio test (SPRT). Further, Non-Patent Document 1 discloses a matrix approximation calculation method in the case of performing multi-class classification in the method based on SPRT disclosed in Patent Document 2.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0004] [ ]

Non-Patent Document 1

[0005] In endoscopic examinations, when detecting lesions from images taken, there are two types of lesion detection methods: one based on a fixed number of images, and another based on a variable number of images, as described in Patent Document 2. The method based on a fixed number of images can detect lesions with high accuracy even when there is no change in the image, but it has the problem of being susceptible to noise, including blurring and distortion. On the other hand, the lesion detection method based on a variable number of images, as described in Patent Document 2, is less susceptible to instantaneous noise and can detect easily identifiable lesions early, but it has the problem of potentially delaying or missing lesion detection when there is no change in the image.

[0006] One of the purposes of this disclosure is to provide an image processing device, an image processing method, and a storage medium that can suitably perform lesion detection in endoscopic images, in view of the above-mentioned problems. [Means for solving the problem]

[0007] One embodiment of an image processing apparatus is: An acquisition means for acquiring endoscopic images of a subject taken by an imaging unit installed in the endoscope, A variation detection means for detecting the degree of variation in the endoscopic image, A selection means that selects either a first model that performs inferences about the lesion of the subject based on a predetermined number of endoscopic images based on the degree of variation, or a second model that performs inferences about the lesion based on a variable number of endoscopic images. A lesion detection means for detecting the lesion based on the selected model, which is the first or second model, This is an image processing device that has the following features.

[0008] One aspect of the image processing method is: Computers Endoscopic images of the subject are acquired by the imaging unit installed in the endoscope. The degree of variation in the aforementioned endoscopic image is detected, Based on the degree of variation, one of the following is selected: a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, and a second model that performs inferences about the lesions based on a variable number of endoscopic images. Based on the selected model, which is either the first or second model, the lesion is detected. This is an image processing method.

[0009] program One aspect of this is, Endoscopic images of the subject are acquired by the imaging unit installed in the endoscope. The degree of variation in the aforementioned endoscopic image is detected, Based on the degree of variation, one of the following is selected: a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, and a second model that performs inferences about the lesions based on a variable number of endoscopic images. The computer is instructed to perform a process to detect the lesion based on the selected model, which is either the first or second model. program That is the case. [Effects of the Invention]

[0010] One example of the effects of this disclosure is that it becomes possible to suitably perform lesion detection in endoscopic images. [Brief explanation of the drawing]

[0011] [Figure 1] This shows the general configuration of an endoscopic examination system. [Figure 2] This shows the hardware configuration of the image processing unit. [Figure 3] This is a functional block diagram of an image processing device. [Figure 4] This shows an example of a display screen shown by a display device during an endoscopic examination. [Figure 5]This is a specific example showing the relationship between a time-series variation score calculated from the processing time when the acquisition of an endoscopic image is started and a selection model determined according to the variation score. [Figure 6] This is a graph showing the transition of the first score during the period when the first model is determined as the selection model. [Figure 7] This is a graph showing the transition of the second score during the period when the second model is determined as the selection model. [Figure 8] This is an example of a flowchart executed by an image processing apparatus in the first embodiment. [Figure 9] (A) This is a graph showing the transition of the first score from the processing time when the acquisition of an endoscopic image is started in the first specific example. (B) This is a graph showing the transition of the second score in the first specific example. [Figure 10] (A) This is a graph showing the transition of the first score S1 in the second specific example. (B) This is a graph showing the transition of the second score in the second specific example. [Figure 11] (A) This is a graph showing the transition of the first score from the processing time when the acquisition of an endoscopic image is started. (B) This is a graph showing the transition of the second score. [Figure 12] This is an example of a flowchart showing the details of the lesion detection process based on the first model in step S15 in the second embodiment. [Figure 13] This is an example of a flowchart showing the details of the lesion detection process based on the second model in step S14 in the second embodiment. [Figure 14] This is a block diagram of an image processing apparatus in the third embodiment. [Figure 15] This is an example of a flowchart showing the processing procedure in the third embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of an image processing apparatus, an image processing method, and a storage medium will be described while referring to the drawings.

[0013] <First Embodiment> (1-1) System Configuration Figure 1 shows a schematic configuration of the endoscopic examination system 100. The endoscopic examination system 100 detects areas of a subject suspected of having a lesion (lesion site) for examiners such as physicians performing examinations or treatments using an endoscope, and presents the detection results. In this way, the endoscopic examination system 100 can support the decision-making of examiners such as physicians, such as determining the treatment plan for the person being examined. As shown in Figure 1, the endoscopic examination system 100 mainly comprises an image processing device 1, a display device 2, and an endoscope scope 3 connected to the image processing device 1.

[0014] The image processing device 1 acquires images (also called "endoscopic image Ia") taken by the endoscope scope 3 in a time series from the endoscope scope 3 and displays a screen based on the endoscopic image Ia on the display device 2. The endoscopic image Ia is an image taken at a predetermined frame period during at least one of the insertion or withdrawal process of the endoscope scope 3 into the patient. In this embodiment, the image processing device 1 analyzes the endoscopic image Ia to detect endoscopic image Ia that includes lesion sites and displays information regarding the detection results on the display device 2.

[0015] The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1.

[0016] The endoscope scope 3 mainly consists of an operating section 36 for the examiner to make predetermined inputs, a flexible shaft 37 that is inserted into the organ to be imaged by the patient, a tip section 38 that incorporates an imaging unit such as a miniature image sensor, and a connecting section 39 for connecting to the image processing device 1.

[0017] Note that the configuration of the endoscopic examination system 100 shown in Figure 1 is just one example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may be composed of multiple devices.

[0018] Hereafter, we will describe the procedure in colonoscopy as a representative example, but the subject may not be limited to the large intestine, but may also be the esophagus or stomach. Furthermore, examples of endoscopes covered in this disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenal endoscopes, small bowel endoscopes, colonoscopes, capsule endoscopes, thoracos (a) Head and neck: Pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: Esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, achalasia, submucosal tumors of the esophagus, benign tumors of the esophagus (c) Stomach: Stomach cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor (d) Duodenum: Duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: Small intestinal cancer, small intestinal neoplastic diseases, small intestinal inflammatory diseases, small intestinal vascular diseases (f) Large intestine: Colon cancer, colon neoplastic disease, colon inflammatory disease, colon polyps, colon polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0019] (1-2) Hardware configuration Figure 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, memory 12, interface 13, input unit 14, light source unit 15, and sound output unit 16. Each of these elements is connected via a data bus 19.

[0020] The processor 11 executes predetermined processes by running programs and other data stored in memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0021] Memory 12 consists of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory that stores information necessary for processing by the image processing device 1. Memory 12 may also include external storage devices such as hard disks connected to or built into the image processing device 1, or it may include storage media such as removable flash memory. Memory 12 stores programs for the image processing device 1 to execute each of the processes in this embodiment.

[0022] Furthermore, the memory 12 functionally includes a first model information storage unit D1 for storing first model information and a second model information storage unit D2 for storing second model information. The first model information includes information regarding the parameters of the first model used by the image processing device 1 for detecting lesion sites. The first model information may also further include information indicating the calculation results of the lesion site detection process using the first model. Furthermore, the second model information includes information regarding the parameters of the second model used by the image processing device 1 for detecting lesion sites. Furthermore, the second model information may also further include information indicating the calculation results of the lesion site detection process using the second model.

[0023] The first model is a model that performs inferences about lesions in a subject based on a fixed predetermined number of endoscopic images (which may be one or multiple images). Specifically, the first model is a model that has learned the relationship between a predetermined number of endoscopic images or their features, which are input to the lesion determination model, and the determination result regarding the lesion site in said endoscopic images. In other words, the first model is a model that has been trained to output a determination result regarding the lesion site in said endoscopic images when input data consisting of a predetermined number of endoscopic images or their features is input. In this embodiment, the determination result regarding the lesion site output by the first model includes at least a score regarding the presence or absence of a lesion site in the endoscopic image, and this score will hereafter be referred to as the "first score S1". For the sake of explanation, the higher the first score S1, the higher the confidence that a lesion site exists in the target endoscopic image. The above-mentioned determination result regarding the lesion site may further include information indicating the location or region (area) of the lesion site in the endoscopic image.

[0024] The first model is, for example, a deep learning model that includes a convolutional neural network in its architecture. For example, the first model may be a Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, DeepLab, etc. The first model information storage unit D1 contains various parameters necessary to construct the first model, such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter. The first model is pre-trained based on a pair of input data, which is an endoscopic image or its features, that conforms to the input format of the first model, and ground truth data that shows the correct determination result regarding the lesion site in the endoscopic image.

[0025] The second model is a model that performs inferences about lesions in a subject based on a variable number of endoscopic images. Specifically, the second model is a machine learning model that learns the relationship between a variable number of endoscopic images or their features and the determination result regarding the lesion site in the endoscopic image. In other words, the second model is a model that has been trained to output a determination result regarding the lesion site in the endoscopic image when input data consisting of a variable number of endoscopic images or their features is input. In this embodiment, the "determination result regarding the lesion site" includes at least a score regarding the presence or absence of a lesion site in the endoscopic image, and this score will hereafter be called the "second score S2". For the sake of explanation, the higher the second score S2, the higher the confidence that a lesion site exists in the target endoscopic image. The second model can be, for example, a model based on SPRT as described in Patent Document 2. A specific example of the second model based on SPRT will be described later. The second model information storage unit D2 stores various parameters necessary to constitute the second model.

[0026] Furthermore, in addition to the first model information and the second model information, the memory 12 stores various information such as parameters necessary for lesion detection processing. At least a portion of the information stored in the memory 12 may be stored by an external device other than the image processing device 1. In this case, the external device may be one or more server devices that can communicate data with the image processing device 1 via a communication network or directly.

[0027] Interface 13 performs interface operations between the image processing device 1 and an external device. For example, interface 13 supplies display information "Ib" generated by the processor 11 to the display device 2. Interface 13 also supplies light generated by the light source unit 15 to the endoscope scope 3. Interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope scope 3 to the processor 11. Interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or it may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc.

[0028] The input unit 14 generates input signals based on operations performed by the examiner. The input unit 14 can be, for example, a button, touch panel, remote controller, or voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope scope 3. The light source unit 15 may also incorporate a pump for supplying water or air to the endoscope scope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0029] (1-3) Overview of lesion detection process Next, an overview of the lesion detection process (lesion detection process) performed by the image processing device 1 will be described. In general terms, the image processing device 1 switches between lesion detection processing based on the first model and lesion detection processing based on the second model based on the degree of variation in the endoscopic image Ia over time. As a result, the image processing device 1 selects the appropriate model from the first and second models depending on the situation and performs lesion detection with high accuracy.

[0030] Figure 3 is a functional block diagram of the image processing device 1. As shown in Figure 3, the processor 11 of the image processing device 1 functionally includes an endoscopic image acquisition unit 30, a variation detection and selection unit 31, a first score calculation unit 32, a second score calculation unit 33, a lesion detection unit 34, and a display control unit 35. In Figure 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to Figure 3. The same applies to the diagrams of other functional blocks described later.

[0031] The endoscopic image acquisition unit 30 acquires endoscopic images Ia captured by the endoscope scope 3 via the interface 13 at predetermined intervals according to the frame cycle of the endoscope scope 3, and supplies the acquired endoscopic images Ia to the fluctuation detection / selection unit 31 and the display control unit 35. Then, each of the subsequent processing units performs the processing described below, using the time interval at which the endoscopic image acquisition unit 30 acquires the endoscopic images as the period. Hereafter, this time for each frame cycle will also be referred to as the "processing time".

[0032] The variation detection and selection unit 31 calculates a score (also called the "variation score") that represents the degree of variation between the endoscopic image Ia at time index t, which represents the current processing time (also called the "current processing image"), and the endoscopic image Ia acquired at the immediately preceding time (i.e., time index "t-1") (also called the "past image"). The variation score becomes larger the greater the degree of variation between the current processing image and the past image. For example, the variation detection and selection unit 31 calculates an arbitrary similarity index based on the comparison between images (i.e., the comparison between images) as the variation score. Examples of similarity indexes in this case include the correlation coefficient, the SSIM (Structural SIMilarity) index, the PSNR (Peak Signal-to-Noise Ratio) index, and the squared error between corresponding pixels.

[0033] Furthermore, the variation detection and selection unit 31 determines, based on the variation score, whether to select the first model or the second model as the model to be used to determine the presence or absence of a lesion site (also called the "selection model").

[0034] Here, the lesion detection process based on the first model has the advantage of being able to detect lesions even under conditions where the second score S2 (log-likelihood ratio described later) based on the second model does not increase easily, when there is no temporal change in the endoscopic image Ia (i.e., the variation score is relatively low). On the other hand, the lesion detection process based on the second model has the advantage of being robust to instantaneous noise and being able to quickly detect easily identifiable lesion sites.

[0035] Taking the above into consideration, if the variation score is below a predetermined threshold (also called the "variation threshold"), the variation detection and selection unit 31 sets the selected model to the first model and supplies a command to calculate the first score S1 and the endoscopic image Ia to the first score calculation unit 32. As a result, lesion detection processing based on the first model, which enables accurate lesion detection even when the time change of the endoscopic image Ia is small, is executed. On the other hand, if the variation score is greater than the variation threshold, the variation detection and selection unit 31 sets the selected model to the second model and notifies the command to calculate the second score S2 and the endoscopic image Ia to the second score calculation unit 33. As a result, lesion detection processing based on the second model, which enables accurate lesion detection even when the time change of the endoscopic image Ia is large, is executed.

[0036] When the first score calculation unit 32 receives a command to calculate the first score S1 from the fluctuation detection / selection unit 31, it calculates the first score S1 based on the first model information storage unit D1. In this case, the first score calculation unit 32 obtains the first score S1 output by the first model by inputting the endoscopic image Ia into the first model configured by referring to the first model information storage unit D1. If the first model is a model that outputs the first score S1 based on one endoscopic image Ia, the first score calculation unit 32 calculates the first score S1 at the current processing time by inputting, for example, the endoscopic image Ia obtained at the current processing time into the first model. If the first model is a model that outputs the first score S1 based on multiple endoscopic images Ia, the first score calculation unit 32 may calculate the first score S1 at the current processing time by inputting, for example, a combination of the endoscopic image Ia obtained at the current processing time and the endoscopic image Ia obtained at a past processing time into the first model. The first score calculation unit 32 may also calculate a first score S1 by averaging (i.e., performing a moving average) the score obtained at past processing times and the score obtained at the current processing time. The first score calculation unit 32 supplies the calculated first score S1 to the lesion detection unit 34.

[0037] When the second score calculation unit 33 receives a command to calculate the second score S2 from the variation detection / selection unit 31, it calculates the second score S2, which indicates the likelihood of the presence of a lesion, based on the second model information storage unit D2 and the variable number of time-series endoscopic images Ia obtained to date. In this case, the second score calculation unit 33 determines the second score S2 for each processing time based on the likelihood ratio for the time-series endoscopic images Ia calculated using the second model based on SPRT. Here, the "likelihood ratio for the time-series endoscopic images Ia" refers to the ratio of the likelihood of the presence of a lesion in the time-series endoscopic images Ia to the likelihood of the absence of a lesion in the time-series endoscopic images Ia. In this embodiment, as an example, the likelihood ratio increases as the likelihood of the presence of a lesion increases. A specific example of the method for calculating the second score S2 using the second model based on SPRT will be described later. The second score calculation unit 33 supplies the calculated second score S2 to the lesion detection unit 34.

[0038] Furthermore, at least one of the first or second model may include an architecture for a feature extractor that converts endoscopic images Ia into feature quantities represented in a predetermined-dimensional feature space (specifically, feature vectors or tensor data of order three or higher). The feature extractor described above may be a deep learning model having an architecture such as a convolutional neural network. In this case, the feature extractor has undergone machine learning in advance, and the parameters obtained through learning are stored in memory 12 or the like. The feature extractor may also extract feature quantities that represent the relationships between time-series data based on any method for calculating the relationships between time-series data, such as LSTM (Long Short-Term Memory). In addition, if the feature extractor is included in only one of the first or second model, the variation detection / selection unit 31 and the first score calculation unit 32 may exchange feature data indicating the feature quantities output by the feature extractor and share feature data.

[0039] Furthermore, the feature extractor may be a model independent of the first and second models. In this case, for example, the feature extraction unit that converts the endoscopic image Ia into feature quantities based on the feature extractor is provided between the variation detection / selection unit 31 and the first score calculation unit 32 and the second score calculation unit 33. The feature extraction unit configures the feature extractor based on parameters pre-stored in the memory 12, etc., and obtains the feature quantities output by the feature extractor by inputting the endoscopic image Ia into the feature extractor. The feature data indicating the feature quantities is then supplied to either the first score calculation unit 32 or the second score calculation unit 33, depending on the model selection result by the variation detection / selection unit 31. In this case, the first score calculation unit 32 obtains the first score S1 output by the first model by inputting the feature quantities of the endoscopic image Ia into the first model configured by referring to the first model information storage unit D1. Furthermore, the second score calculation unit 33 calculates the second score S2 output by the second model by inputting the feature quantities of the time-series endoscopic image Ia into the second model, which is configured by referring to the second model information storage unit D2.

[0040] The lesion detection unit 34 performs lesion detection processing (i.e., determination of the presence or absence of a lesion) in the endoscopic image Ia based on the first score S1 supplied from the first score calculation unit 32 or the second score S2 supplied from the second score calculation unit 33. In this case, the lesion detection unit 34 determines the presence or absence of a lesion based on the score (first score S1 or second score S2) based on the model (first model or second model) defined as the selection model by the variation detection / selection unit 31. Specific examples of the processing of the lesion detection unit 34 will be described later. The lesion detection unit 34 supplies the lesion detection result to the display control unit 35.

[0041] The display control unit 35 generates display information Ib based on the endoscopic image Ia and the lesion detection results supplied from the lesion detection unit 34, and supplies the display information Ib to the display device 2 via the interface 13, thereby causing the display device 2 to display information regarding the endoscopic image Ia and the lesion detection results from the lesion detection unit 34. The display control unit 35 may also cause the display device 2 to further display information regarding the selection model used in the lesion detection process (including information regarding the score calculated by the selection model).

[0042] Figure 4 shows an example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 35 of the image processing device 1 outputs display information Ib, which is generated based on the endoscopic image Ia acquired by the endoscopic image acquisition unit 30 and the lesion detection results from the lesion detection unit 34, to the display device 2. The display control unit 35 transmits the endoscopic image Ia and the display information Ib to the display device 2, thereby displaying the above-mentioned display screen on the display device 2. In the example of the display screen shown in Figure 4, the display control unit 35 of the image processing device 1 provides a real-time image display area 71, a lesion detection result display area 72, and a processing details display area 73 on the display screen.

[0043] Here, the display control unit 35 displays a moving image representing the latest endoscopic image Ia in the real-time image display area 71. Furthermore, in the lesion detection result display area 72, the display control unit 35 displays the lesion detection result from the lesion detection unit 34. At the time the display screen shown in Figure 4 is displayed, the lesion detection unit 34 has determined that a lesion exists, so the display control unit 35 displays a text message indicating that there is a high probability that a lesion is present in the lesion detection result display area 72. Alternatively, the display control unit 35 may output a sound (including voice) from the sound output unit 16 to notify that there is a high probability that a lesion is present, in addition to displaying the text message indicating that there is a high probability that a lesion is present in the lesion detection result display area 72.

[0044] Furthermore, in the processing details display area 73, the display control unit 35 displays the fluctuation score, the selected model, and a graph showing the transition of the score calculated by the selected model. Here, as an example, the display control unit 35 represents the fluctuation score with two values, "high" or "low," and here, since the fluctuation score is below the fluctuation threshold, it is indicated that the fluctuation score is "low." Also, since the fluctuation score is below the fluctuation threshold and lesion detection processing based on the first model was being performed, the display control unit 35 displays that lesion detection processing based on the first model is currently being executed. In addition, as a graph showing the transition of the score of the selected model, the display control unit 35 displays a score transition graph showing the change in the first score S1 from the start of the endoscopic examination to the present, along with a dashed line indicating the reference value (first score threshold Sth1, described later) for determining the presence or absence of a lesion from the first score S1.

[0045] Here, the components of the endoscopic image acquisition unit 30, the variation detection and selection unit 31, the first score calculation unit 32, the second score calculation unit 33, the lesion detection unit 34, and the display control unit 35 can be realized, for example, by the processor 11 executing a program. Alternatively, the necessary programs may be recorded on any non-volatile storage medium and installed as needed to realize each component. At least a portion of these components may be realized not only by software programs, but also by any combination of hardware, firmware, and software. Furthermore, at least a portion of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the program composed of the above components may be realized using this integrated circuit. Also, at least a portion of each component may be composed of an ASSP (Application Specific Standard Produce), ASIC (Application Specific Integrated Circuit), or quantum processor (quantum computer control chip). Thus, each component may be realized by various hardware. The same applies to other embodiments described later. Furthermore, each of these components may be realized through the collaboration of multiple computers, for example, using cloud computing technology.

[0046] (1-4) Example of calculating the second score Next, we will explain an example of calculating the second score S2 using a second model based on SPRT.

[0047] The second score calculation unit 33 calculates the likelihood ratio for the most recent "N" (N is an integer of 2 or more) endoscopic images Ia for each processing time, and determines the second score S2 based on the likelihood ratio (also called the "integrated likelihood ratio") obtained by integrating the likelihood ratios calculated at the current processing time and past processing times. The second score S2 may be the integrated likelihood ratio itself, or it may be a function that includes the integrated likelihood ratio as a variable. Hereafter, for the sake of explanation, the second model will be assumed to include a likelihood ratio calculation model, which is a processing unit that calculates the likelihood ratio, and a score calculation model, which is a processing unit that calculates the second score S2 from the likelihood ratio.

[0048] The likelihood ratio calculation model is a model trained to output the likelihood ratio for N endoscopic images Ia when given feature data for those N images Ia. The likelihood ratio calculation model may be a deep learning model, any other machine learning model, or a statistical model. In this case, for example, the second model information storage unit D2 stores the trained parameters of the second model, including the likelihood ratio calculation model. If the likelihood ratio calculation model is composed of a neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter are pre-stored in the second model information storage unit D2. The second score calculation unit 33 can also obtain the likelihood ratio using the likelihood ratio calculation model from fewer than N endoscopic images Ia, even if the number of acquired endoscopic images Ia is less than N. The second score calculation unit 33 may store the acquired likelihood ratio in the second model information storage unit D2.

[0049] Next, we will explain the score calculation model included in the second model. The current processing time is defined as time index "t" when a predetermined start time is time index "1", and the feature quantities of any endoscopic image Ia to be processed are defined as "x". iLet i=1,...,t. The "start time" represents the first processing time of past processing times considered in the calculation of the second score S2. In this case, the combined likelihood ratio for the binary classification between class "C1" which includes the lesion site and class "C0" in which endoscopic image Ia does not include the lesion site is expressed by the following equation (1).

[0050]

number

[0051] In equation (1), the time index t, which represents the current processing time, increases with the passage of time, so the length of the time series of endoscopic images Ia used to calculate the integrated likelihood ratio (i.e., the number of frames) is variable. Thus, by using the integrated likelihood ratio based on equation (1), the second score calculation unit 33 has the first advantage of being able to calculate the second score S2 considering a variable number of endoscopic images Ia. In addition, by using the integrated likelihood ratio based on equation (1), the second advantage is that time-dependent features can be classified, and the third advantage is that a second score S2 that does not degrade easily even with difficult-to-distinguish data can be suitably calculated. The second score calculation unit 33 may store the integrated likelihood ratio and the second score S2 calculated at each processing time in the second model information storage unit D2.

[0052] Furthermore, if the second score calculation unit 33 reaches a predetermined threshold where the second score S2 is negative, it may determine that there is no lesion, initialize the second score S2 and the time index t to 0, and restart the calculation of the second score S2 based on the endoscopic image Ia obtained from the next processing time.

[0053] (1-5) Processing by the lesion detection unit Next, the specific method by which the lesion detection unit 34 determines the presence or absence of a lesion site will be explained. The lesion detection unit 34 determines the presence or absence of a lesion site based on the score of the selection model determined by the variation detection and selection unit 31.

[0054] Figure 5 is a specific example showing the relationship between the time-series variation score calculated from the processing time "t0" when the acquisition of endoscopic image Ia began, and the selection model determined according to that variation score.

[0055] In the example shown in Figure 5, since the variation score is below the variation threshold during the period from processing time t0 to processing time t10, the variation detection and selection unit 31 determines the first model as the selected model for that period, and the lesion detection unit 34 determines the presence or absence of a lesion based on the first score S1.

[0056] On the other hand, during the period from processing time t10 to processing time t20, the fluctuation score becomes greater than the fluctuation threshold, so the fluctuation detection / selection unit 31 determines the second model as the selected model for that period, and the lesion detection unit 34 determines the presence or absence of a lesion based on the second score S2. Furthermore, during the period after processing time t20, the fluctuation score becomes below the fluctuation threshold again, so the fluctuation detection / selection unit 31 determines the first model as the selected model for that period, and the lesion detection unit 34 determines the presence or absence of a lesion based on the first score S1.

[0057] Figure 6 is a graph showing the transition of the first score S1 during the period from processing time t0, when the first model is determined as the selected model, to processing time t10.

[0058] If the selected model is the first model, the lesion detection unit 34 compares the first score S1 with the threshold for the first score S1 (also called the "first score threshold Sth1") at each processing time. The lesion detection unit 34 then determines that a lesion exists if the first score S1 exceeds the first score threshold Sth1 for more than a predetermined number of consecutive times (also called the "threshold count Mth").

[0059] Hereafter, the number of consecutive times the first score S1 exceeds the first score threshold Sth1 will be referred to as the "threshold overrun count M". The first score threshold Sth1 and the threshold count Mth are pre-stored values, for example, in memory 12.

[0060] In the example shown in Figure 6, the lesion detection unit 34 determines at processing time "t1" that the first score S1 exceeds the first score threshold Sth1, and starts counting the number of consecutive occurrences M exceeding the threshold. At processing time "t2", it determines that the number of consecutive occurrences M exceeding the threshold exceeds the threshold count Mth. Therefore, in this case, the lesion detection unit 34 determines that a lesion exists in the endoscopic image Ia obtained during the period from processing time t1 to t2.

[0061] Figure 7 is a graph showing the transition of the second score S2 during the period from processing time t10, when the second model is determined as the selected model, to processing time t20.

[0062] If the selected model is the second model, the lesion detection unit 34 compares the second score S2 obtained at each processing time with a threshold for the second score S2 (also called the "second score threshold Sth2"). If the second score S2 is greater than the second score threshold Sth2, the lesion detection unit 34 determines that a lesion exists. On the other hand, if the second score S2 is less than a predetermined negative threshold, the lesion detection unit 34 determines that no lesion exists. The predetermined threshold is set, for example, to have an absolute value that is the same negative value as the second score threshold Sth2 (i.e., -Sth2). The second score threshold Sth2 and the predetermined threshold are pre-stored in memory 12, etc.

[0063] In the example shown in Figure 7, the lesion detection unit 34 calculates a second score S2 at each processing time based on the endoscopic image Ia obtained after processing time t10. At processing time t11, since the second score S2 falls below "-Sth2", which corresponds to the predetermined threshold mentioned above, the lesion detection unit 34 determines that there are no lesions in the endoscopic image Ia obtained between processing time t10 and processing time t11. In this case, the lesion detection unit 34 resets the second score S2 to 0 and recalculates the second score S2 based on the endoscopic image Ia obtained after processing time t11. Furthermore, at processing time t12, since the second score S2 falls below "-Sth2" again, the lesion detection unit 34 determines that there are no lesions in the endoscopic image Ia obtained between processing time t11 and processing time t12, and resets the second score S2. On the other hand, after starting the recalculation of the second score S2 at processing time t12, at processing time t13, the second score S2 exceeded the second score threshold Sth2, and therefore the lesion detection unit 34 determined that a lesion was present in the endoscopic image Ia obtained during the period from processing time t12 to processing time t13.

[0064] Here, we will provide a supplementary explanation of the advantages and disadvantages of the lesion detection process using the first model based on a convolutional neural network and the lesion detection process using the second model based on SPRT.

[0065] When using a convolutional neural network-based model for lesion detection, the presence or absence of lesion detection is determined by comparing the threshold number of supercontinuous counts M with the threshold number of counts Mth, in order to improve specificity. This type of lesion detection has the advantage of being able to detect lesions even under conditions where the log-likelihood ratio calculated in the second model based on SPRT does not increase easily, such as when there is no temporal change in the endoscopic image Ia. On the other hand, compared to the lesion detection processing based on the second model, it is more susceptible to noise (including blur and smudges), and even for easily identifiable lesion sites, a larger number of endoscopic images Ia are required to detect the lesion. In contrast, the second model based on SPRT is robust against instantaneous noise and can quickly detect easily identifiable lesion sites, but the log-likelihood ratio does not increase easily when there is little temporal change in the endoscopic image Ia, and a larger number of endoscopic images Ia may be required to detect the lesion. Taking the above into consideration, in this embodiment, the image processing device 1 adaptively switches between the first model and the second model as the selected model used for lesion detection processing based on the variation score. This allows the selected model to be set to the first model when lesion detection processing based on the first model is effective, and to be set to the second model when lesion detection processing based on the second model is effective, thereby suitably improving lesion detection accuracy.

[0066] (1-6) Processing flow Figure 8 is an example of a flowchart executed by the image processing device 1 in the first embodiment.

[0067] First, the endoscopic image acquisition unit 30 of the image processing device 1 acquires an endoscopic image Ia (step S11). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope scope 3 via the interface 13. The display control unit 35 then performs processing such as displaying the endoscopic image Ia acquired in step S11 on the display device 2.

[0068] Next, the variation detection and selection unit 31 of the image processing device 1 calculates a variation score based on the current processing image, which is the endoscopic image Ia obtained in step S11 of the current processing time, and the past image, which is the endoscopic image Ia obtained in step S11 of the previous processing time (step S12). Then, the variation detection and selection unit 31 determines whether the variation score is greater than the variation threshold (step S13).

[0069] Then, if the variation detection / selection unit 31 determines that the variation score is greater than the variation threshold (step S13; Yes), the image processing device 1 performs lesion detection processing based on the second model (step S14). In this case, the second score calculation unit 33 calculates the second score S2 based on the second model, and the lesion detection unit 34 determines the presence or absence of a lesion based on the comparison result between the second score S2 and the second score threshold Sth2.

[0070] On the other hand, if the variation detection / selection unit 31 determines that the variation score is below the variation threshold (step S13; No), the image processing device 1 performs lesion detection processing based on the first model (step S15). In this case, the first score calculation unit 32 calculates the first score S1 based on the first model and determines the presence or absence of a lesion based on the comparison result between the threshold over-consecutive count M, which is the number of consecutive times the first score S1 exceeds the first score threshold Sth1, and the threshold count Mth.

[0071] The image processing device 1 then determines whether the endoscopic examination is complete or not (step S16). For example, the image processing device 1 determines that the endoscopic examination is complete when it detects a predetermined input to the input unit 14 or the operation unit 36. If the image processing device 1 determines that the endoscopic examination is complete (step S16; Yes), it terminates the flowchart process. On the other hand, if the image processing device 1 determines that the endoscopic examination is not complete (step S16; No), it returns to step S11.

[0072] (1-7) Variation Next, a modified version of the first embodiment described above will be explained. The following modifications may be combined in any way.

[0073] (Extreme Variation 1-1) Instead of calculating the variation score by directly comparing the current processed image with past images, the variation detection and selection unit 31 may calculate the variation score based on the similarity between the features of the current processed image and the features of past images. In this way, the variation detection and selection unit 31 can also calculate a variation score that indicates the substantial similarity between the current processed image and past images, and suitably determine whether or not there is variation.

[0074] (Variations 1-2) The image processing device 1 may process the video, which is composed of endoscopic images Ia generated during the endoscopic examination, after the examination.

[0075] For example, at any time after the examination, if the image processing device 1 is specified based on user input from the input unit 14 or the like, it repeatedly performs the flowchart shown in Figure 8 on the time-series endoscopic images Ia that constitute the image until it is determined that the target image has finished.

[0076] <Second Embodiment> (2-1) overview In the second embodiment, when the image processing device 1 performs lesion detection processing based on the first model, it changes the threshold count Mth used in the lesion detection processing based on the second score S2 output by the second model. Here, the threshold count Mth is a parameter that defines the conditions for determining that a lesion has been detected based on the first score S1, and the image processing device 1 changes the threshold count Mth to relax the above conditions as the confidence level of the presence of a lesion indicated by the second score S2 increases. Similarly, when performing lesion detection processing based on the second model, the image processing device 1 changes the second score threshold Sth2 used in the lesion detection processing based on the first score S1 output by the first model. Here, the second score threshold Sth2 is a parameter that defines the conditions for determining that a lesion has been detected based on the second score S2, and the image processing device 1 changes the second score threshold Sth2 to relax the above conditions as the confidence level of the presence of a lesion indicated by the first score S1 increases.

[0077] Hereafter, components of the endoscopic examination system 100 similar to those in the first embodiment will be denoted by the same reference numerals as in the first embodiment, and their descriptions will be omitted. Furthermore, the hardware configuration of the image processing device 1 according to the second embodiment is the same as the hardware configuration of the image processing device 1 shown in Figure 2, and the functional block configuration of the processor 11 of the image processing device 1 according to the second embodiment is the same as the functional block configuration shown in Figure 3. Note that the threshold count Mth and the second score threshold Sth2 are examples of "parameters used for lesion detection based on the selection model".

[0078] (2-2) Lesion detection processing based on the first model First, we will specifically explain the case where lesion detection processing based on the first model is performed.

[0079] The first score calculation unit 32 and the second score calculation unit 33 calculate the first score S1 and the second score S2 at each processing time, regardless of the selection model determined by the fluctuation detection / selection unit 31, and supply the calculation results to the lesion detection unit 34. The lesion detection unit 34 then determines that a lesion exists if the first score S1 exceeds the first score threshold Sth1 for more consecutive times than the threshold count Mth. On the other hand, the lesion detection unit 34 decreases the threshold count Mth if the second score S2 becomes greater than the second score threshold Sth2. In this way, the lesion detection unit 34 relaxes the condition for determining the existence of a lesion based on the first score S1 when the existence of a lesion is suspected based on the second score S2 output by the second model. As a result, the lesion detection unit 34 can more accurately determine the presence or absence of a lesion in the lesion detection process based on the first model.

[0080] Here, specific examples of lesion detection processing based on the first model according to the second embodiment (first example, second example) will be explained with reference to Figures 9(A) to 10(B).

[0081] Figure 9(A) is a graph showing the progression of the first score S1 from the processing time "t30" when the acquisition of the endoscopic image Ia began in the first specific example, and Figure 9(B) is a graph showing the progression of the second score S2 from the processing time t30 in the first specific example.

[0082] In this case, at each processing time after processing time t30, the lesion detection unit 34 compares the first score S1 obtained at each processing time with the first score threshold Sth1, and the second score S2 with the second score threshold Sth2. The lesion detection unit 34 then determines at processing time t31 that the first score S1 exceeds the first score threshold Sth1 and starts counting the number of consecutive threshold exceedances M. At processing time t32, it determines that the number of consecutive threshold exceedances M exceeds the threshold count Mth. Therefore, in this case, the lesion detection unit 34 determines that a lesion exists in the endoscopic image Ia obtained at processing times t31 to t32. On the other hand, at processing time t30 and later, the lesion detection unit 34 determines that the second score S2 is less than or equal to the second score threshold Sth2, and keeps the threshold count Mth fixed even at processing time t30 and later.

[0083] Figure 10(A) is a graph showing the progression of the first score S1 from processing time "t40" in the second specific example, and Figure 10(B) is a graph showing the progression of the second score S2 from processing time "t40" in the second specific example.

[0084] In the second specific example, at each processing time after processing time t40, the lesion detection unit 34 compares the first score S1 obtained at each processing time with the first score threshold Sth1, and the second score S2 with the second score threshold Sth2. Then, during the period from processing time "t41" to processing time "t42", the number of consecutive threshold exceedances M increases because the first score S1 exceeds the first score threshold Sth1. On the other hand, since the number of consecutive threshold exceedances M does not reach the initial threshold count Mth, and the first score S1 becomes less than or equal to the first score threshold Sth1 after processing time t42, the lesion detection unit 34 determines that no lesion exists during the above period.

[0085] Meanwhile, at processing time "t43", the lesion detection unit 34 determines that the second score S2 is greater than the second score threshold Sth2 and sets the threshold count Mth to a predetermined relaxation value smaller than the initial value (i.e., a value in which the condition for determining the presence of a lesion is relaxed from the initial value). The initial value of the threshold count Mth and the relaxation value of the threshold count Mth are stored in advance in memory 12, for example.

[0086] Subsequently, after processing time "t44", the number of consecutive threshold occurrences M increases because the first score S1 exceeds the first score threshold Sth1. Then, from processing time t44 to processing time "t45", the first score S1 exceeds the first score threshold Sth1, and the number of consecutive threshold occurrences M exceeds the relaxation value of the threshold count Mth. Therefore, the lesion detection unit 34 determines that a lesion exists during the period from processing time t44 to processing time t45.

[0087] In this way, the lesion detection unit 34 can suitably relax the condition for determining the presence of a lesion when the second score S2 based on the second model reaches the second score threshold Sth2, thereby improving the accuracy of the lesion detection process based on the first model. Furthermore, if an easily discernible lesion is present, the relaxation of the above condition allows for rapid lesion detection with fewer endoscopic images Ia. In this case, reducing the number of endoscopic images Ia required for lesion detection reduces the possibility of instantaneous noise causing the threshold over-continuous count M to be initialized.

[0088] (2-3) Lesion detection processing based on the second model Next, we will specifically explain the case where lesion detection processing based on the second model is performed.

[0089] The first score calculation unit 32 and the second score calculation unit 33, regardless of the selection model determined by the fluctuation detection / selection unit 31, respectively perform the processing to calculate the first score S1 and the second score S2 at each processing time, and supply the calculation results to the lesion detection unit 34. The lesion detection unit 34 then compares the first score S1 with the first score threshold Sth1, and the second score S2 with the second score threshold Sth2 at each processing time. The lesion detection unit 34 then determines that a lesion exists if the second score S2 is greater than the second score threshold Sth2. On the other hand, during the period when the number of consecutive occurrences exceeding the threshold M increases, the lesion detection unit 34 gradually or continuously lowers the second score threshold Sth2 as the number of consecutive occurrences exceeding the threshold M increases (i.e., it relaxes the conditions for considering a lesion to have been detected). As a result, the lesion detection unit 34 can relax the conditions for lesion detection based on the second model in the lesion detection process based on the second model, depending on the situation, and more accurately determine whether or not the first lesion site is present.

[0090] Figure 11(A) is a graph showing the progression of the first score S1 from processing time "t50" when the acquisition of endoscopic image Ia began, and Figure 11(B) is a graph showing the progression of the second score S2 from processing time t50.

[0091] In this case, at each processing time after processing time t50, the lesion detection unit 34 compares the first score S1 obtained at each processing time with the first score threshold Sth1, and the second score S2 with the second score threshold Sth2, respectively. Then, at processing time t51, the lesion detection unit 34 determines that the first score S1 exceeds the first score threshold Sth1 and increases the number of consecutive threshold exceedances M.

[0092] Then, the lesion detection unit 34 changes the second score threshold Sth2 according to the number of consecutive occurrences M after the processing time t51, which is the start time of the period of increasing number of consecutive occurrences M exceeding the threshold. Here, the lesion detection unit 34 continuously decreases the second score threshold Sth2 as the number of consecutive occurrences M exceeding the threshold increases. Then, at processing time t52, which is included in the period of increasing number of consecutive occurrences M exceeding the threshold, the second score S2 becomes greater than the second score threshold Sth2, so the lesion detection unit 34 determines that a lesion exists at processing time t52.

[0093] In this way, the lesion detection unit 34 reduces the second score threshold Sth2 as the number of consecutive threshold occurrences M increases, thereby suitably relaxing the conditions for lesion detection related to the second score S2 based on the second model and enabling accurate lesion detection.

[0094] (2-4) Processing flow In the second embodiment, the image processing device 1 executes the flowchart shown in Figure 8, which was described in the first embodiment. In this case, in the lesion detection process based on the first model in step S15, the flowchart shown in Figure 12, which will be described later, is executed, and in the lesion detection process based on the second model in step S14, the flowchart shown in Figure 13, which will be described later, is executed.

[0095] Figure 12 is an example flowchart showing the details of the lesion detection process based on the first model in step S15 in the second embodiment.

[0096] In Figure 12, the second score calculation unit 33 calculates a second score S2 based on a variable number of endoscopic images Ia (step S20). In this case, for example, the second score calculation unit 33 calculates the second score S2 based on a variable number of endoscopic images Ia or their feature data acquired at the current processing time and past processing times, and a second model configured based on the second model information storage unit D2. In parallel with step S20, the first score calculation unit 32 calculates a first score S1 based on a predetermined number of endoscopic images Ia (step S24). In this case, for example, the first score calculation unit 32 calculates the first score S1 based on a predetermined number of endoscopic images Ia or their feature data acquired at the current processing time (and past processing times), and a first model configured based on the first model information storage unit D1.

[0097] After step S20 is executed, the lesion detection unit 34 determines whether the second score S2 is greater than the second score threshold Sth2 (step S21). If the second score S2 is greater than the second score threshold Sth2 (step S21; Yes), the lesion detection unit 34 sets the threshold count Mth to a relaxation value smaller than the initial value (step S22). On the other hand, if the second score S2 is less than or equal to the second score threshold Sth2 (step S21; No), the lesion detection unit 34 sets the threshold count Mth to the initial value (step S23).

[0098] Furthermore, after step S24 is executed, the lesion detection unit 34 determines whether the first score S1 is greater than the first score threshold Sth1 (step S25). If the first score S1 is greater than the first score threshold Sth1 (step S25; Yes), the lesion detection unit 34 increases the threshold over-consecutive count M by 1 (step S26). The initial value of the threshold over-consecutive count M is assumed to be 0. On the other hand, if the first score S1 is less than or equal to the first score threshold Sth1 (step S25; No), the lesion detection unit 34 sets the threshold over-consecutive count M to its initial value of 0 (step S27). Then, the flowchart processing ends.

[0099] Next, after the completion of step S22 or step S23 and step S26, the lesion detection unit 34 determines whether the number of consecutive occurrences M exceeding the threshold is greater than the threshold number Mth (step S28). If the number of consecutive occurrences M exceeding the threshold is greater than the threshold number Mth (step S28; Yes), the lesion detection unit 34 determines that a lesion site exists and notifies the system that a lesion site has been detected by display or sound output (step S29). On the other hand, if the number of consecutive occurrences M exceeding the threshold is less than or equal to the threshold number Mth (step S28; No), the process of the flowchart is terminated.

[0100] Figure 13 is an example flowchart showing the details of the lesion detection process based on the second model in step S14 in the second embodiment.

[0101] First, the second score calculation unit 33 calculates a second score S2 based on a variable number of endoscopic images Ia (step S31). In this case, for example, the second score calculation unit 33 calculates the second score S2 based on a variable number of endoscopic images Ia or their characteristic data acquired at the current processing time and past processing times, and a second model configured based on the second model information storage unit D2. Also, in parallel with step S20, the first score calculation unit 32 calculates a first score S1 based on a predetermined number of endoscopic images Ia (step S32). In this case, for example, the first score calculation unit 32 calculates the first score S1 based on a predetermined number of endoscopic images Ia or their characteristic data acquired at the current processing time (and past processing times), and a first model configured based on the first model information storage unit D1.

[0102] After step S32 is executed, the lesion detection unit 34 determines whether the first score S1 is greater than the first score threshold Sth1 (step S33). If the first score S1 is greater than the first score threshold Sth1 (step S33; Yes), the lesion detection unit 34 increases the threshold over-consecutive count M by 1 (step S34). The initial value of the threshold over-consecutive count M is assumed to be 0. On the other hand, if the first score S1 is less than or equal to the first score threshold Sth1 (step S33; No), the lesion detection unit 34 sets the threshold over-consecutive count M to its initial value of 0 (step S35).

[0103] Then, after step S34 or step S35 is executed, the lesion detection unit 34 determines a second score threshold Sth2, which is a threshold to be compared with the second score S2, based on the number of consecutive occurrences M that exceeds the threshold (step S36). In this case, the lesion detection unit 34 refers to, for example, a pre-stored formula or lookup table, and the larger the number of consecutive occurrences M that exceeds the threshold, the smaller the second score threshold Sth2 becomes.

[0104] After steps S31 and S36 are executed, the lesion detection unit 34 determines whether the second score S2 is greater than the second score threshold Sth2 (step S37). If the second score S2 is greater than the second score threshold Sth2 (step S37; Yes), the lesion detection unit 34 determines that a lesion exists and notifies the system that a lesion has been detected by displaying a message or outputting an audible signal (step S38). On the other hand, if the second score S2 is less than or equal to the second score threshold Sth2 (step S37; No), the system returns to step S31.

[0105] (2-5) Variation Next, a modified version of the second embodiment described above will be explained. The following modifications may be combined in any way.

[0106] (Variation 2-1) In the lesion detection process based on the first model, the lesion detection unit 34 switched the threshold count Mth from an initial value to a relaxed value when the second score S2 exceeded the second score threshold Sth2. On the other hand, the lesion detection unit 34 is not limited to this embodiment, and may gradually or continuously decrease the threshold count Mth (i.e., relax the conditions for determining the presence of a lesion) as the second score S2 increases.

[0107] In this case, for example, correspondence information such as an expression or lookup table showing the relationship between each possible second score S2 and the appropriate threshold count Mth for each second score S2 is pre-stored in memory 12 or the like. Then, in the lesion detection process based on the first model, the lesion detection unit 34 determines the threshold count Mth based on the second score S2 and the aforementioned correspondence information. In this embodiment as well, the lesion detection unit 34 can set the threshold count Mth according to the second score S2 and perform accurate lesion detection.

[0108] (Variation 2-2) In the lesion detection process based on the first model, the lesion detection unit 34 may change the first score threshold Sth1 based on the second score S2, instead of changing the threshold count Mth based on the second score S2, or in addition to changing the threshold count Mth based on the second score S2. In this case, for example, the lesion detection unit 34 may decrease the first score threshold Sth1 in stages or continuously as the second score S2 increases. This embodiment also allows the lesion detection unit 34 to suitably relax the conditions for lesion detection based on the first model and accurately perform lesion detection.

[0109] (Variations 2-3) In the lesion detection process based on the first model, the image processing device 1 may start the calculation of the second score S2 and the process of changing the threshold count Mth when it determines that predetermined conditions based on the first score S1 have been met.

[0110] For example, after the start of lesion detection processing based on the first model, the image processing device 1 does not calculate the second score S2 using the second score calculation unit 33. If it determines that the first score S1 exceeds the first score threshold Sth1, the second score calculation unit 33 starts calculating the second score S2 and changes the threshold count Mth (or the first score threshold Sth1) according to the second score S2, similar to the embodiment described above. On the other hand, if the image processing device 1 determines that the first score S1 has become less than or equal to the first score threshold Sth1 after starting to calculate the second score S2 using the second score calculation unit 33, the second score calculation unit 33 stops calculating the second score S2 again. Note that the "predetermined conditions" are not limited to the condition that the first score S1 is greater than the first score threshold Sth1, but may be any condition that indicates an increased probability of the presence of a lesion. For example, such conditions include the condition that the first score S1 is greater than a predetermined threshold that is less than the first score threshold Sth1, the condition that the increase of the first score S1 per unit time (i.e., the derivative of the first score S1) is greater than or equal to a predetermined value, and the condition that the threshold number of supercontinuous counts M is greater than or equal to a predetermined value.

[0111] Furthermore, if the image processing device 1 starts calculating the second score S2 when predetermined conditions are met, it may calculate the second score S2 retrospectively to a past processing time and change the threshold count Mth (or the first score threshold Sth1) based on the second score S2. In this case, the image processing device 1 may, for example, store the endoscopic image Ia or its characteristic data obtained at a past processing time in the memory 12, and the second score calculation unit 33 may calculate the second score S2 at a past processing time based on the endoscopic image Ia or its characteristic data and change the threshold count Mth (or the first score threshold Sth1) based on the second score S2.

[0112] According to this modified example, the image processing device 1 can limit the period for calculating the second score S2 in the lesion detection process based on the first model, thereby suitably reducing the computational load.

[0113] (Modification 2-4) In lesion detection processing based on the second model, the image processing device 1 may start calculating the first score S1 based on the first model and modifying the second score threshold Sth2 when it determines that predetermined conditions based on the second score S2 have been met.

[0114] For example, after the start of lesion detection processing based on the second model, the image processing device 1 does not calculate the first score S1 using the first score calculation unit 32. Instead, if the second score S2 is greater than a predetermined threshold (e.g., 0) that is smaller than the second score threshold Sth2, the first score calculation unit 32 starts calculating the first score S1 and changes the second score threshold Sth2 in the same manner as in the above embodiment, according to the number of consecutive occurrences M of the threshold. On the other hand, after the image processing device 1 starts calculating the first score S1 using the first score calculation unit 32, if it determines that the second score S2 has fallen below a predetermined threshold, the first score calculation unit 32 stops calculating the first score S1 again. Note that the "predetermined condition" is not limited to the condition that the second score S2 is greater than a predetermined threshold, but may be any condition that indicates an increased probability of the presence of a lesion. For example, such conditions include the condition that the increase in the second score S2 per unit time (i.e., the derivative of the first score S1) is greater than or equal to a predetermined value.

[0115] Furthermore, if the image processing device 1 starts calculating the first score S1 when predetermined conditions are met, it may calculate the first score S1 retrospectively to past processing times and change the second score threshold Sth2 based on the first score S1. In this case, the image processing device 1 may, for example, store the endoscopic image Ia or its characteristic data obtained at past processing times in the memory 12, and the first score calculation unit 32 may calculate the first score S1 at past processing times based on the endoscopic image Ia or its characteristic data and change the second score threshold Sth2 at past processing times based on the first score S1. In this case, the image processing device 1 compares the second score S2 and the second score threshold Sth2 at each past processing time to determine whether or not there is a lesion.

[0116] According to this modified example, the image processing device 1 can limit the period for calculating the second score S2 in the lesion detection process based on the second model, thereby suitably reducing the computational load.

[0117] (Variations 2-5) The image processing device 1 may process the video, which is composed of endoscopic images Ia generated during the endoscopic examination, after the examination.

[0118] <Third Embodiment> Figure 14 is a block diagram of the image processing device 1X in the third embodiment. The image processing device 1X includes an acquisition means 30X, a variation detection means 311X, a selection means 312X, and a lesion detection means 34X. The image processing device 1X may be composed of multiple devices.

[0119] The acquisition means 30X acquires endoscopic images of the subject taken by the imaging unit provided in the endoscope. In this case, the acquisition means 30X may acquire the endoscopic images generated by the imaging unit immediately, or it may acquire endoscopic images that have been previously generated by the imaging unit and stored in a storage device at a predetermined timing. The acquisition means 30X can be, for example, the endoscopic image acquisition unit 30 in the first or second embodiment.

[0120] The variation detection means 311X detects the degree of variation in the endoscopic image. The selection means 312X selects either a first model that performs inferences about the lesion of the subject based on a predetermined number of endoscopic images, or a second model that performs inferences about the lesion based on a variable number of endoscopic images, based on the degree of variation. The variation detection means 311X and the selection means 312X can be, for example, the variation detection / selection unit 31 in the first or second embodiment.

[0121] The lesion detection means 34X detects lesions based on a selected model, which is either a first or second model. The lesion detection means 34X can be, for example, a first score calculation unit 32, a second score calculation unit 33, and a lesion detection unit 34 in the first or second embodiment.

[0122] Figure 15 is an example of a flowchart showing the processing procedure in the third embodiment. First, the acquisition means 30X acquires endoscopic images of the subject taken by the imaging unit provided on the endoscope (step S41). The variation detection means 311X detects the degree of variation in the endoscopic images (step S42). The selection means 312X selects either a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, or a second model that performs inferences about the lesions based on a variable number of endoscopic images, based on the degree of variation (step S43). The lesion detection means 312X detects lesions based on the selected model, which is either the first model or the second model (step S44).

[0123] According to the third embodiment, the image processing device 1X can accurately detect lesion sites present in the endoscopic image.

[0124] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable medium and supplied to a computer, such as a processor. Non-transitory computer-readable mediums include various types of tangible storage mediums. Examples of non-transitory computer-readable mediums include magnetic storage mediums (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage mediums (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to the computer by various types of transient computer-readable mediums. Examples of transient computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable mediums can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0125] In addition, some or all of the above embodiments (including modifications, the same applies hereinafter) may also be described as follows, but are not limited to the following.

[0126] [Note 1] An acquisition means for acquiring endoscopic images of a subject taken by an imaging unit installed in the endoscope, A variation detection means for detecting the degree of variation in the endoscopic image, A selection means that selects either a first model that performs inferences about the lesion of the subject based on a predetermined number of endoscopic images based on the degree of variation, or a second model that performs inferences about the lesion based on a variable number of endoscopic images. A lesion detection means for detecting the lesion based on the selected model, which is the first or second model, An image processing device having [Note 2] The image processing apparatus according to Appendix 1, wherein the selection means selects the first model as the selected model when the degree of variation is less than or equal to a predetermined threshold, and selects the second model as the selected model when the degree of variation is greater than the threshold. [Note 3] The image processing apparatus according to Appendix 1, wherein the variation detection means calculates the degree of variation by determining the similarity between the endoscopic image obtained at the current processing time and the endoscopic image obtained at the processing time immediately preceding the current processing time. [Note 4] The first model is an image processing device as described in Appendix 1, which includes a convolutional neural network in its architecture. [Note 5] The second model is an image processing apparatus described in Appendix 1, which is based on SPRT. [Note 6] The image processing apparatus according to Appendix 1, wherein the lesion detection means changes the parameters used for detecting the lesion based on the selected model based on a non-selected model which is a first model or a second model that is not the selected model. [Note 7] The aforementioned parameters are parameters that define the conditions for determining that the lesion has been detected. The image processing apparatus according to Appendix 6, wherein the lesion detection means modifies the parameters to relax the conditions as the confidence level of the presence of the lesion, as indicated by the score calculated by the non-selection model, increases. [Note 8] The image processing apparatus according to Appendix 6 or 7, wherein the lesion detection means determines that predetermined conditions based on the score calculated by the selected model have been met, and then starts calculating the score by the non-selected model. [Note 9] The image processing apparatus according to Appendix 1, further comprising output control means for displaying or outputting as sound information related to the detection result of the lesion by the lesion detection means and information related to the selected model. [Note 10] The image processing apparatus according to Appendix 9, wherein the output control means outputs information regarding the detection result of the lesion and information regarding the selection model in order to support the examiner's decision-making. [Note 10] Computers Endoscopic images of the subject are acquired by the imaging unit installed in the endoscope. The degree of variation in the aforementioned endoscopic image is detected, Based on the degree of variation, one of the following is selected: a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, and a second model that performs inferences about the lesions based on a variable number of endoscopic images. Based on the selected model, which is either the first or second model, the lesion is detected. Image processing methods. [Note 11] Endoscopic images of the subject are acquired by the imaging unit installed in the endoscope. The degree of variation in the aforementioned endoscopic image is detected, Based on the degree of variation, one of the following is selected: a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, and a second model that performs inferences about the lesions based on a variable number of endoscopic images. A storage medium storing a program that causes a computer to perform a process to detect the lesion based on the selected model, which is either the first or second model.

[0127] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made as understood by those skilled in the art within the scope of the present invention. That is, the present invention includes the full disclosure, including the claims, and of course, various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, the above-mentioned patent and non-patent disclosures are incorporated herein by reference. [Explanation of symbols]

[0128] 1. 1X Image Processing Device 2 Display device 3 Endoscope 11 processors 12 memory 13 Interfaces 14 Input section 15 Light source section 16. Sound output section 100 Endoscopy System

Claims

1. An acquisition means for acquiring endoscopic images of a subject taken by an imaging unit installed in the endoscope, A variation detection means for detecting the degree of variation in the endoscopic image, A selection means that selects either a first model that performs inferences about the lesion of the subject based on a predetermined number of endoscopic images based on the degree of variation, or a second model that performs inferences about the lesion based on a variable number of endoscopic images, A lesion detection means for detecting the lesion based on the selected model, which is the first or second model, An image processing device having

2. The image processing apparatus according to claim 1, wherein the selection means selects the first model as the selected model when the degree of variation is less than or equal to a predetermined threshold, and selects the second model as the selected model when the degree of variation is greater than the threshold.

3. The image processing apparatus according to claim 1, wherein the variation detection means calculates the degree of variation by determining the similarity between the endoscopic image obtained at the current processing time and the endoscopic image obtained at the processing time immediately preceding the current processing time.

4. The image processing apparatus according to claim 1, wherein the first model is a deep learning model that includes a convolutional neural network in its architecture.

5. The image processing apparatus according to claim 1, wherein the second model is a model based on SPRT.

6. The image processing apparatus according to claim 1, wherein the lesion detection means changes the parameters used for detecting the lesion based on the selected model based on a non-selected model which is a first model or a second model that is not the selected model.

7. The aforementioned parameters are parameters that define the conditions for determining that the lesion has been detected. The image processing apparatus according to claim 6, wherein the lesion detection means modifies the parameters to relax the conditions as the confidence level of the presence of the lesion, as indicated by the score calculated by the non-selection model, increases.

8. The image processing apparatus according to claim 6 or 7, wherein the lesion detection means determines that predetermined conditions based on the score calculated by the selected model have been met, and then starts calculating a score by the non-selected model.

9. The image processing apparatus according to claim 1, further comprising output control means for displaying or outputting as sound information related to the detection result of the lesion by the lesion detection means and information related to the selected model.

10. The image processing apparatus according to claim 9, wherein the output control means outputs information regarding the detection result of the lesion and information regarding the selected model in order to support the examiner's decision-making.

11. Computers Endoscopic images of the subject are acquired by the imaging unit installed in the endoscope. The degree of variation in the aforementioned endoscopic image is detected, Based on the degree of variation, one of the following is selected: a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, and a second model that performs inferences about the lesions based on a variable number of endoscopic images. Based on the selected model, which is either the first or second model, the lesion is detected. Image processing methods.

12. Endoscopic images of the subject are acquired by the imaging unit installed in the endoscope. The degree of variation in the aforementioned endoscopic image is detected, Based on the degree of variation, one of the following is selected: a first model that performs inferences about the subject's lesions based on a predetermined number of endoscopic images, and a second model that performs inferences about the lesions based on a variable number of endoscopic images. A program that causes a computer to perform a process to detect the lesion based on the selected model, which is either the first model or the second model.

Images