Image processing apparatus, image processing method, and program

Abstract

An image processing device 1X comprises an acquisition means 30X and a lesion detection means 34X. The acquisition means 30X acquires an endoscopic image obtained by imaging a subject using an imaging unit provided in an endoscope. The lesion detection means 34X detects a lesion on the basis of a selected model, which is selected from a first model for making an inference about a lesion of a subject on the basis of a predetermined number of endoscope images, and a second model for making an inference about a lesion of the subject on the basis of a variable number of endoscope images. The lesion detection means 34X also changes parameters for use in the detection of a lesion based on the selected model on the basis of a non-selected model which is the first model or the second model that is not the selected model.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that process images acquired during an endoscopic examination.

[0002] Conventionally, endoscopic systems that display images of the inside of organ lumens have been known. For example, Patent Literature 1 discloses a learning method for a learning model that outputs information about a lesion site contained in endoscopic image data when the endoscopic image data generated by an imaging device is input. Furthermore, Patent Literature 2 discloses a classification method for classifying sequential data using a technique that applies a sequential probability ratio test (SPRT). Furthermore, Non-Patent Literature 1 discloses a matrix approximation method for multi-class classification in the SPRT-based technique disclosed in Patent Literature 2.

[0003] International Publication WO2020 / 003607 International Publication WO2020 / 194497

[0004] Miyagawa Taiki, and Akinori F. Ebihara. "The Power of Log-Sum-Exp: Sequential Density Ratio Matrix Estimation for Speed-Accuracy Optimization." International Conference on Machine Learning. PMLR, 2021.

[0005] When detecting lesions from images captured during endoscopic examinations, there are lesion detection methods based on a fixed, predetermined number of images and lesion detection methods based on a variable number of images, as described in Patent Document 2. While lesion detection methods based on a predetermined number of images can detect lesions with high accuracy even when there is no change in the image, they have the problem of being susceptible to noise, including blurring and blur. Furthermore, lesion detection methods based on a variable number of images, as described in Patent Document 2, are less susceptible to momentary noise and can detect easily identifiable lesions early, but they have the problem of potentially delaying or overlooking lesion detection when there is no change in the image.

[0006] In view of the above-mentioned problems, one object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of suitably detecting lesions in endoscopic images.

[0007] One aspect of the image processing device comprises: an acquisition means for acquiring endoscopic images of a subject captured by an imaging unit provided in an endoscope; and a lesion detection means for detecting the lesion based on a selection model selected from a first model for making inferences about lesions in the subject based on a predetermined number of the endoscopic images, and a second model for making inferences about the lesion based on a variable number of the endoscopic images, wherein the lesion detection means is an image processing device that changes parameters used to detect the lesion based on the selection model based on a non-selection model, which is the first model or the second model that is not the selection model.

[0008] One aspect of the image processing method is an image processing method in which a computer acquires endoscopic images of a subject taken by an imaging unit provided in an endoscope, detects the lesion based on a selected model selected from a first model that makes inferences about lesions in the subject based on a predetermined number of the endoscopic images and a second model that makes inferences about the lesion based on a variable number of the endoscopic images, and changes parameters used to detect the lesion based on the selected model based on a non-selected model that is the first model or the second model that is not the selected model.

[0009] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute the following process: acquire endoscopic images of a subject taken by an imaging unit provided in an endoscope; detect the lesion based on a selection model selected from a first model that makes an inference about a lesion in the subject based on a predetermined number of the endoscopic images; and a second model that makes an inference about the lesion based on a variable number of the endoscopic images; and change parameters used to detect the lesion based on the selection model based on a non-selection model that is the first model or the second model that is not the selection model.

[0010] One example of the effect of the present disclosure is that lesion detection in endoscopic images can be performed favorably.

[0011] 8 shows a schematic configuration of an endoscopic examination system. FIG. 9 shows a hardware configuration of an image processing device. FIG. 10 is a functional block diagram of the image processing device. FIG. 11 shows an example of a display screen displayed by a display device during endoscopic examination. (A) A graph showing the progression of a first score from processing time t0 when acquisition of an endoscopic image starts in a first specific example. (B) A graph showing the progression of a second score from processing time t0 in a first specific example. (A) A graph showing the progression of a first score from processing time t0 in a second specific example. (B) A graph showing the progression of a second score from processing time t0 in a second specific example. An example of a flowchart executed by an image processing device in the first embodiment. (A) A graph showing the progression of a first score from processing time t0 in a second embodiment, and FIG. 11(B) is a graph showing the progression of a second score from processing time t0 in a second embodiment. An example of a flowchart executed by an image processing device in the second embodiment. An example of a flowchart executed by an image processing device in the third embodiment. A block diagram of an image processing device in a fourth embodiment. An example of a flowchart executed by an image processing device in the fourth embodiment.

[0012] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0013] <First Embodiment> (1-1) System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. The endoscopic examination system 100 detects a region of a subject suspected of having a lesion (lesion region) and presents the detection results to an examiner such as a doctor who performs an examination or treatment using an endoscope. In this way, the endoscopic examination system 100 can support the examiner such as a doctor in making decisions, such as determining a treatment plan for the subject of the examination. As shown in Fig. 1, the endoscopic examination system 100 mainly includes an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1.

[0014] The image processing device 1 acquires images (also referred to as "endoscopic images Ia") captured by the endoscope 3 in time series from the endoscope 3, and displays a screen based on the endoscopic images Ia on the display device 2. The endoscopic images Ia are images captured at a predetermined frame rate during at least one of the steps of inserting or ejecting the endoscope 3 into the subject. In this embodiment, the image processing device 1 analyzes the endoscopic images Ia to detect the endoscopic images Ia that include a lesion site, and displays information related to the detection results on the display device 2.

[0015] The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the image processing device 1 .

[0016] The endoscope 3 mainly comprises an operation unit 36 ​​for the examiner to input predetermined information, a flexible shaft 37 that is inserted into the subject's organ to be photographed, a tip 38 that incorporates an imaging unit such as a micro-imaging element, and a connection unit 39 for connecting to the image processing device 1.

[0017] 1 is an example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may be configured from multiple devices.

[0018] Hereinafter, processing in an endoscopic examination of the large intestine will be described as a representative example, but the subject is not limited to the large intestine, and the esophagus or stomach may also be the subject of the examination. Examples of endoscopes that are subject to the present disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, angioscopes, and epidural endoscopes. Examples of pathological conditions at lesion sites that are subject to detection in the present disclosure include (a) to (f) below. (a) Head and neck: pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor (c) Stomach: gastric cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor (d) Duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease (f) Large intestine: large intestine cancer, large intestine neoplastic disease, large intestine inflammatory disease, large intestine polyp, large intestine polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0019] (1-2) Hardware Configuration Fig. 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0020] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0021] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs for the image processing device 1 to execute each process in this embodiment.

[0022] Furthermore, memory 12 functionally includes a first model information storage unit D1 that stores first model information and a second model information storage unit D2 that stores second model information. The first model information includes information on parameters of the first model used by image processing device 1 to detect a lesion site. The first model information may further include information indicating a calculation result of the lesion site detection process using the first model. The second model information includes information on parameters of the second model used by image processing device 1 to detect a lesion site. The second model information may further include information indicating a calculation result of the lesion site detection process using the second model.

[0023] The first model is a model that performs inference regarding a lesion in a subject based on a fixed, predetermined number of endoscopic images (which may be one or multiple). Specifically, the first model is a model that has learned the relationship between a predetermined number of endoscopic images or their feature values ​​input to the lesion determination model and a determination result regarding a lesion site in the endoscopic images. In other words, the first model is a model that has been trained to output a determination result regarding a lesion site in an endoscopic image when input data that is a predetermined number of endoscopic images or their feature values ​​is input. In this embodiment, the determination result regarding a lesion site output by the first model includes at least a score (index value) regarding the presence or absence of a lesion site in the endoscopic image. This score will hereinafter be referred to as the "first score S1." For ease of explanation, a higher first score S1 indicates a higher degree of certainty that a lesion site exists in the target endoscopic image. Note that the above-described determination result regarding a lesion site may further include information indicating the location or area of ​​the lesion site in the endoscopic image.

[0024] The first model is, for example, a deep learning model that includes a convolutional neural network in its architecture. For example, the first model may be a Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, DeepLab, or the like. The first model information storage unit D1 includes various parameters necessary for configuring the first model, such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter. The first model is trained in advance based on a combination of endoscopic images or their features, which are input data conforming to the input format of the first model, and correct answer data indicating the correct answer determination result regarding the lesion site in the endoscopic image.

[0025] The second model is a model that performs inference regarding lesions in a subject based on a variable number of endoscopic images. Specifically, the second model is a model that performs machine learning to determine the relationship between a variable number of endoscopic images or their feature values ​​and a determination result regarding a lesion location in the endoscopic images. In other words, the second model is a model that is trained to output a determination result regarding a lesion location in the endoscopic images when input data that is a variable number of endoscopic images or their feature values ​​is input. In this embodiment, the "determination result regarding the lesion location" includes at least a score regarding the presence or absence of a lesion location in the endoscopic image, and this score will hereinafter be referred to as the "second score S2." For ease of explanation, the higher the second score S2, the higher the certainty that a lesion location exists in the target endoscopic image. The second model can be, for example, a model based on the SPRT method described in Patent Document 2. Specific examples of the second model based on the SPRT method will be described later. The second model information storage unit D2 stores various parameters required to configure the second model.

[0026] Furthermore, in addition to the first model information and the second model information, various information such as parameters necessary for the lesion detection process is stored in the memory 12. At least a part of the information stored in the memory 12 may be stored in an external device other than the image processing device 1. In this case, the external device may be one or more server devices capable of data communication with the image processing device 1 via a communication network or the like or by direct communication.

[0027] The interface 13 acts as an interface between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0028] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0029] (1-3) Overview of Lesion Detection Processing Next, an overview of the lesion detection processing (lesion detection processing) performed by the image processing device 1 will be described. In general terms, when performing lesion detection based on the first score S1 output by the first model, the image processing device 1 changes the parameters used for the lesion detection based on the second score S2 output by the second model. Specifically, the above parameters define the conditions for determining that a lesion has been detected based on the first score S1, and the image processing device 1 changes the parameters to relax the above conditions as the confidence level of the presence of a lesion indicated by the second score S2 increases. In this way, the image processing device 1 performs accurate lesion detection by taking advantage of the advantages of both the first model and the second model, and presents the detection results. In the first embodiment, the first model is an example of a "selection model," and the second model is an example of a "non-selection model."

[0030] Fig. 3 is a functional block diagram of the image processing device 1. As shown in Fig. 3, the processor 11 of the image processing device 1 functionally includes an endoscopic image acquisition unit 30, a feature extraction unit 31, a first score calculation unit 32, a second score calculation unit 33, a lesion detection unit 34, and a display control unit 35. Note that in Fig. 3, blocks between which data is exchanged are connected by solid lines, but the combination of blocks between which data is exchanged is not limited to that shown in Fig. 3. The same applies to other functional block diagrams described later.

[0031] The endoscopic image acquisition unit 30 acquires endoscopic images Ia captured by the endoscope 3 via the interface 13 at predetermined intervals in accordance with the frame period of the endoscope 3, and supplies the acquired endoscopic images Ia to the feature extraction unit 31 and the display control unit 35. Then, each processing unit in the subsequent stages performs the processing described below, using the time interval at which the endoscopic image acquisition unit 30 acquires the endoscopic images as a period. Hereinafter, the time for each frame period will also be referred to as the "processing time."

[0032] The feature extraction unit 31 converts the endoscopic image Ia supplied from the endoscopic image acquisition unit 30 into feature quantities (more specifically, feature vectors or third-order or higher tensor data) expressed in a feature space of a predetermined dimension. In this case, for example, the feature extraction unit 31 configures a feature extractor based on parameters pre-stored in the memory 12 or the like, and acquires feature quantities output by the feature extractor by inputting the endoscopic image Ia to the feature extractor. Here, the feature extractor may be a deep learning model having an architecture such as a convolutional neural network. In this case, the feature extractor is subjected to machine learning in advance, and parameters obtained by learning are pre-stored in the memory 12 or the like. Note that the feature extractor may extract feature quantities representing relationships between time-series data based on any method for calculating relationships between time-series data, such as LSTM (Long Short Term Memory). The feature extraction unit 31 then supplies feature data representing the generated feature quantities to the first score calculation unit 32 and the second score calculation unit 33.

[0033] The feature extractor described above may be incorporated into at least one of the first model and the second model. For example, if the first model includes the architecture of the feature extractor, the first score calculation unit 32 inputs the endoscopic image Ia to the first model, and then supplies feature amount data indicating the feature amounts generated by the feature extractor in the first model to the second score calculation unit 33 as an output of the intermediate layer of the first model. In this case, the feature extraction unit 31 may not be provided.

[0034] The first score calculation unit 32 calculates the first score S1 based on the first model information storage unit D1 and the feature data supplied from the feature extraction unit 31. In this case, the first score calculation unit 32 acquires the first score S1 output from the first model by inputting the feature data supplied from the feature extraction unit 31 into the first model configured with reference to the first model information storage unit D1. Note that if the first model is a model that outputs the first score S1 based on one endoscopic image Ia, the first score calculation unit 32 may calculate the first score S1 at the current processing time by, for example, inputting the feature data supplied from the feature extraction unit 31 at the current processing time to the first model. Also, if the first model is a model that outputs the first score S1 based on multiple endoscopic images Ia, the first score calculation unit 32 may calculate the first score S1 at the current processing time by, for example, inputting a combination of the feature data supplied from the feature extraction unit 31 at the current processing time and feature data previously supplied to the first model. The first score calculation unit 32 may also calculate the first score S1 by averaging (i.e., performing a moving average) the scores obtained at past processing times and the scores obtained at the current processing time. The first score calculation unit 32 supplies the calculated first score S1 to the lesion detection unit 34.

[0035] The second score calculation unit 33 calculates a second score S2 indicating the likelihood of the presence of a lesion based on the second model information storage unit D2 and feature data corresponding to a variable number of time-series endoscopic images Ia obtained up to now. In this case, the second score calculation unit 33 determines the second score S2 based on the likelihood ratio for the time-series endoscopic images Ia calculated for each processing time using the second model based on SPRT. Here, the "likelihood ratio for the time-series endoscopic images Ia" refers to the ratio between the likelihood of the presence of a lesion in the time-series endoscopic images Ia and the likelihood of the absence of a lesion in the time-series endoscopic images Ia. In this embodiment, as an example, the likelihood ratio increases as the likelihood of the presence of a lesion increases. A specific example of a method for calculating the second score S2 using the second model based on SPRT will be described later. The second score calculation unit 33 supplies the calculated second score S2 to the lesion detection unit 34.

[0036] The lesion detection unit 34 detects a lesion in the endoscopic image Ia (i.e., determines whether a lesion exists) based on the first score S1 supplied from the first score calculation unit 32 and the second score S2 supplied from the second score calculation unit 33. In this case, the lesion detection unit 34 changes the threshold value that defines the condition for determining that a lesion has been detected based on the first score S1, based on the second score S2. A specific example of the processing by the lesion detection unit 34 will be described later. The lesion detection unit 34 supplies the lesion detection result to the display control unit 35.

[0037] The display control unit 35 generates display information Ib based on the endoscopic image Ia and the lesion detection result supplied from the lesion detection unit 34, and supplies the display information Ib to the display device 2 via the interface 13, thereby causing the display device 2 to display information related to the endoscopic image Ia and the lesion detection result by the lesion detection unit 34. The display control unit 35 may also cause the display device 2 to display information related to the first score S1 calculated by the first score calculation unit 32 and the second score S2 calculated by the second score calculation unit 33.

[0038] 4 shows an example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 35 of the image processing device 1 outputs display information Ib generated based on the endoscopic image Ia acquired by the endoscopic image acquisition unit 30 and the lesion detection result by the lesion detection unit 34, etc., to the display device 2. The display control unit 35 transmits the endoscopic image Ia and the display information Ib to the display device 2, thereby causing the display device 2 to display the above-mentioned display screen. In the example of the display screen shown in FIG. 4, the display control unit 35 of the image processing device 1 provides a real-time image display area 71, a lesion detection result display area 72, and a score transition display area 73 on the display screen.

[0039] Here, the display control unit 35 displays a moving image representing the latest endoscopic image Ia in the real-time image display area 71. Furthermore, the display control unit 35 displays the lesion detection result by the lesion detection unit 34 in the lesion detection result display area 72. Note that, at the time the display screen shown in FIG. 4 is displayed, the lesion detection unit 34 has determined that a lesion site is present, and therefore the display control unit 35 is displaying a text message indicating that a lesion is highly likely to be present in the lesion detection result display area 72. Note that, instead of or in addition to displaying the text message indicating that a lesion is highly likely to be present in the lesion detection result display area 72, the display control unit 35 may output a sound (including voice) notifying that a lesion is highly likely to be present from the sound output unit 16.

[0040] In addition, in the score transition display area 73, the display control unit 35 displays a score transition graph showing the progress of the first score S1 from the start of the endoscopic examination to the present time, together with a dotted line indicating the reference value (first score threshold Sth1 described later) for determining the presence or absence of a lesion from the first score S1.

[0041] Here, each of the components of the endoscopic image acquisition unit 30, the feature extraction unit 31, the first score calculation unit 32, the second score calculation unit 33, the lesion detection unit 34, and the display control unit 35 can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording the necessary program on an arbitrary non-volatile storage medium and installing it as needed. Note that at least some of these components are not limited to being realized by software programs, but may also be realized by any combination of hardware, firmware, and software. Furthermore, at least some of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0042] (1-4) Example of Calculation of Second Score Next, an example of calculation of the second score S2 using the second model based on SPRT will be described.

[0043] The second score calculation unit 33 calculates likelihood ratios for the latest "N" (N is an integer greater than or equal to 2) endoscopic images Ia for each processing time, and determines the second score S2 based on a likelihood ratio (also referred to as an "integrated likelihood ratio") obtained by integrating the likelihood ratios calculated at the current processing time and past processing times. Note that the second score S2 may be the integrated likelihood ratio itself, or may be a function including the integrated likelihood ratio as a variable. Hereinafter, for ease of explanation, the second model is assumed to include a likelihood ratio calculation model, which is a processing unit that calculates the likelihood ratios, and a score calculation model, which is a processing unit that calculates the second score S2 from the likelihood ratios.

[0044] The likelihood ratio calculation model is a model trained to output likelihood ratios for N endoscopic images Ia when feature data of the N endoscopic images Ia are input. The likelihood ratio calculation model may be a deep learning model, any other machine learning model, or a statistical model. In this case, for example, the second model information storage unit D2 stores trained parameters of the second model including the likelihood ratio calculation model. When the likelihood ratio calculation model is configured using a neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter are stored in advance in the second model information storage unit D2. Note that even when the number of acquired endoscopic images Ia is less than N, the second score calculation unit 33 can acquire likelihood ratios from fewer than N endoscopic images Ia using the likelihood ratio calculation model. The second score calculation unit 33 may store the acquired likelihood ratios in the second model information storage unit D2.

[0045] Next, a score calculation model included in the second model will be described. When a predetermined start time is set as time index "1", the current processing time is set as time index "t", and the feature amount of an arbitrary endoscopic image Ia to be processed is set as "x ｉ (i=1, . . . , t). The "start time" represents the first processing time of the past processing times considered in calculating the second score S2. In this case, the class "C １ " and class "C" where endoscopic image Ia does not contain a lesion. ０The integrated likelihood ratio for the binary classification of " is expressed by the following equation (1):

[0046] Here, "p" represents the probability of belonging to each class (i.e., the confidence level between 0 and 1). In calculating the term on the right side of equation (1), the likelihood ratio output by the likelihood ratio calculation model can be used.

[0047] In Equation (1), the time index t representing the current processing time increases over time, so the length of the endoscopic image Ia in the time series (i.e., the number of frames) used to calculate the integrated likelihood ratio is variable. Thus, by using the integrated likelihood ratio based on Equation (1), the second score calculation unit 33 can calculate the second score S2 taking into account a variable number of endoscopic images Ia. Furthermore, by using the integrated likelihood ratio based on Equation (1), the second score calculation unit 33 can classify time-dependent features as a second advantage, and can preferably calculate the second score S2 with low accuracy even for difficult-to-distinguish data as a third advantage. The second score calculation unit 33 may store the integrated likelihood ratio and the second score S2 calculated at each processing time in the second model information storage unit D2.

[0048] In addition, when the second score S2 reaches a predetermined threshold value that is a negative value, the second score calculation unit 33 may determine that no lesion area exists, initialize the second score S2 and the time index t to 0, and restart the calculation of the second score S2 based on the endoscopic image Ia obtained from the next processing time.

[0049] (1-5) Processing of the Lesion Detection Unit Next, a specific method for determining the presence or absence of a lesion by the lesion detection unit 34 will be described. The lesion detection unit 34 compares the first score S1 with a threshold for the first score S1 (also referred to as the "first score threshold Sth1") and the second score S2 with a threshold for the second score S2 (also referred to as the "second score threshold Sth2") at each processing time. The lesion detection unit 34 then determines that a lesion exists when the first score S1 exceeds the first score threshold Sth1 a predetermined number of times (also referred to as the "threshold number Mth") in succession. On the other hand, when the second score S2 becomes greater than the second score threshold Sth2, the lesion detection unit 34 reduces the threshold number Mth. In this way, the lesion detection unit 34 relaxes the conditions for determining that a lesion exists based on the first score S1 in a situation where the presence of a lesion is suspected based on the second score S2 output by the second model. This makes it possible to accurately detect the lesion site in both situations where the first model is likely to accurately detect the lesion site and where the second model is likely to accurately detect the lesion site.

[0050] Hereinafter, the number of times that the first score S1 consecutively exceeds the first score threshold Sth1 will be referred to as the "number of consecutive times M that exceeds the threshold." Note that the first score threshold Sth1 and the second score threshold Sth2 each have matching values ​​stored in advance in, for example, the memory 12. The threshold number Mth is a value that varies depending on the second score S2, and an initial value, for example, is stored in advance in the memory 12. The threshold number Mth is an example of a "parameter used for lesion detection based on a selection model."

[0051] Next, a method for determining lesion detection by the lesion detection unit 34 will be described using a first specific example shown in FIGS. 5(A) and 5(B) and a second specific example shown in FIGS. 6(A) and 6(B).

[0052] 5A is a graph showing the progress of the first score S1 from processing time "t0" when acquisition of the endoscopic image Ia begins in the first specific example, and FIG. 5B is a graph showing the progress of the second score S2 from processing time t0 in the first specific example. Note that the first specific example is an example of lesion detection processing in a situation where the accuracy of lesion detection based on the first model is higher than the accuracy of lesion detection based on the second model. For example, such a situation may be one in which the fluctuation of the endoscopic image Ia over time is relatively small.

[0053] In the first specific example, at each processing time after processing time t0, the lesion detection unit 34 compares the first score S1 obtained at each processing time with the first score threshold Sth1, and the second score S2 obtained at each processing time with the second score threshold Sth2. Then, at processing time "t1," the lesion detection unit 34 determines that the first score S1 exceeds the first score threshold Sth1 and starts counting the number of consecutive exceeding-threshold values ​​M. At processing time "t1α," the lesion detection unit 34 determines that the number of consecutive exceeding-threshold values ​​M exceeds the threshold number Mth. Therefore, in this case, the lesion detection unit 34 determines that a lesion exists in the endoscopic image Ia obtained from processing time t1 to t1α. Meanwhile, after processing time t0, the lesion detection unit 34 determines that the second score S2 is equal to or less than the second score threshold Sth2, and keeps the threshold number Mth fixed even after processing time t0.

[0054] In this way, in a situation where the accuracy of lesion detection based on the first model is higher than the accuracy of lesion detection based on the second model, the second score S2 based on the second model does not reach the second score threshold Sth2, but the first score S1 based on the first model stably reaches the first score threshold Sth1. Therefore, in such a situation, the lesion detection unit 34 can accurately perform lesion detection.

[0055] 6A is a graph showing the progress of the first score S1 from processing time t0 in the second specific example, and FIG. 6B is a graph showing the progress of the second score S2 from processing time t0 in the second specific example. Note that the second specific example is an example of lesion detection processing in a situation where the accuracy of lesion detection based on the second model is higher than the accuracy of lesion detection based on the first model. For example, such a situation may include a case where the fluctuation of the endoscopic image Ia over time is relatively large.

[0056] In the second specific example, at each processing time after processing time t0, the lesion detection unit 34 compares the first score S1 obtained at each processing time with the first score threshold Sth1, and the second score S2 obtained at each processing time with the second score threshold Sth2. Then, during the period from processing time "t2" to processing time "t3," the first score S1 exceeds the first score threshold Sth1, and the consecutive exceed-threshold count M increases. Meanwhile, since the consecutive exceed-threshold count M does not exceed the initial threshold count Mth, the first score S1 becomes equal to or less than the first score threshold Sth1 after processing time t3, and the lesion detection unit 34 determines that no lesion is present during the period.

[0057] On the other hand, at processing time "t4", the lesion detection unit 34 determines that the second score S2 is greater than the second score threshold Sth2, and sets the threshold number of times Mth to a predetermined relaxed value that is smaller than the initial value (i.e., a value that relaxes the condition for determining that a lesion exists compared to the initial value). Note that the initial value of the threshold number of times Mth and the relaxed value of the threshold number of times Mth are each stored in advance in the memory 12, for example.

[0058] After that, from processing time "t5" onwards, the first score S1 exceeds the first score threshold Sth1, and therefore the consecutive exceed-threshold count M increases. Then, since the first score S1 exceeds the first score threshold Sth1 from processing time t5 to processing time "t6", and the consecutive exceed-threshold count M becomes greater than the relaxed value of the threshold count Mth, the lesion detection unit 34 determines that a lesion site is present during the period from processing time t5 to processing time t6.

[0059] In this way, in a situation where the lesion detection accuracy based on the second model is higher than that based on the first model, the second score S2 based on the second model reaches the second score threshold Sth2, and the conditions for determining the presence of a lesion can be suitably relaxed. Therefore, even in such a situation, the lesion detection unit 34 can accurately perform lesion detection based on the first model. Furthermore, in the presence of an easily identifiable lesion, relaxing the above-described conditions allows lesion detection to be performed more quickly with fewer endoscopic images Ia. In this case, reducing the number of endoscopic images Ia required for lesion detection reduces the possibility of momentary noise causing the initialization of the threshold-exceeding number M.

[0060] Here, we will provide a supplementary explanation of the advantages and disadvantages of the first model based on a convolutional neural network and the second model based on SPRT when each is used independently for lesion detection.

[0061] When a model based on a convolutional neural network is used for lesion detection, the presence or absence of a lesion is determined by comparing the number of consecutive occurrences exceeding the threshold M with the threshold number Mth to improve specificity. Such lesion detection has the advantage of being able to detect lesions even under conditions in which the log-likelihood ratio calculated using the second model based on SPRT is unlikely to increase, such as when there is no time change in the endoscopic image Ia. On the other hand, compared to lesion detection based on the second model, the second model is more susceptible to noise (including blurring and blurring) and requires a larger number of endoscopic images Ia to detect a lesion, even for lesions that are easily identifiable. In contrast, the second model based on SPRT is more resistant to instantaneous noise and can quickly detect lesions that are easily identifiable. However, when there is little time change in the endoscopic image Ia, the log-likelihood ratio is less likely to increase, and a larger number of endoscopic images Ia may be required to detect a lesion. In this embodiment, these two models are combined to preferably perform lesion detection that enjoys the advantages of both models.

[0062] (1-6) Processing Flow Figure 7 is an example of a flowchart executed by the image processing device 1 in the first embodiment. The image processing device 1 repeatedly executes the processing of this flowchart until the end of the endoscopic examination. Note that, for example, the image processing device 1 determines that the endoscopic examination has ended when it detects a predetermined input to the input unit 14 or the operation unit 36.

[0063] First, the endoscopic image acquisition unit 30 of the image processing device 1 acquires an endoscopic image Ia (step S11). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope 3 via the interface 13. The display control unit 35 also executes processing such as displaying the endoscopic image Ia acquired in step S11 on the display device 2. The feature extraction unit 31 also generates feature data indicating the feature amounts of the acquired endoscopic image Ia.

[0064] Next, the second score calculation unit 33 calculates a second score S2 based on the variable number of endoscopic images Ia (step S12). In this case, for example, the second score calculation unit 33 calculates the second score S2 based on feature data of the variable number of endoscopic images Ia acquired at the current processing time and past processing times and a second model constructed based on the second model information storage unit D2. Furthermore, in parallel with step S12, the first score calculation unit 32 calculates a first score S1 based on a predetermined number of endoscopic images Ia (step S16). In this case, for example, the first score calculation unit 32 calculates the first score S1 based on feature data of the predetermined number of endoscopic images Ia acquired at the current processing time (and past processing times) and a first model constructed based on the first model information storage unit D1.

[0065] After executing step S12, the lesion detection unit 34 determines whether the second score S2 is greater than the second score threshold Sth2 (step S13). If the second score S2 is greater than the second score threshold Sth2 (step S13; Yes), the lesion detection unit 34 sets the threshold number of times Mth to a relaxed value less than the initial value (step S14). On the other hand, if the second score S2 is equal to or less than the second score threshold Sth2 (step S13; No), the lesion detection unit 34 sets the threshold number of times Mth to the initial value (step S15).

[0066] After step S16, the lesion detection unit 34 determines whether the first score S1 is greater than the first score threshold Sth1 (step S17). If the first score S1 is greater than the first score threshold Sth1 (step S17; Yes), the lesion detection unit 34 increases the number of consecutive over-threshold occurrences M by 1 (step S18). The initial value of the number of consecutive over-threshold occurrences M is set to 0. On the other hand, if the first score S1 is equal to or less than the first score threshold Sth1 (step S17; No), the lesion detection unit 34 sets the number of consecutive over-threshold occurrences M to the initial value of 0 (step S19).

[0067] Next, after completing step S14 or step S15 and step S18, the lesion detection unit 34 determines whether the consecutive exceeding-threshold number M is greater than the threshold number Mth (step S20). If the consecutive exceeding-threshold number M is greater than the threshold number Mth (step S20; Yes), the lesion detection unit 34 determines that a lesion is present and notifies the user that a lesion has been detected by at least one of display and sound output (step S21). On the other hand, if the consecutive exceeding-threshold number M is equal to or less than the threshold number Mth (step S20; No), the process returns to step S11.

[0068] (1-7) Modifications Next, modifications of the first embodiment will be described. The following modifications may be combined in any manner.

[0069] (Variation 1-1) When the second score S2 exceeds the second score threshold Sth2, the lesion detection unit 34 switches the threshold number of times Mth from the initial value to a relaxed value. However, the lesion detection unit 34 is not limited to this configuration, and may gradually or continuously decrease the threshold number of times Mth (i.e., relax the conditions for determining that a lesion exists) as the second score S2 increases.

[0070] In this case, correspondence information such as an equation or a lookup table showing the relationship between each possible second score S2 and the threshold number Mth appropriate for each second score S2 is stored in advance in the memory 12, and the lesion detection unit 34 determines the threshold number Mth based on the second score S2 and the correspondence information. Even in this mode, the lesion detection unit 34 sets the threshold number Mth according to the second score S2, and can perform lesion detection that takes advantage of the advantages of both the first model and the second model.

[0071] (Variation 1-2) Instead of or in addition to changing the threshold number of times Mth based on the second score S2, the lesion detection unit 34 may change the first score threshold Sth1 based on the second score S2. In this case, for example, the lesion detection unit 34 may gradually or continuously decrease the first score threshold Sth1 as the second score S2 increases. This also allows the lesion detection unit 34 to appropriately relax the conditions for lesion detection based on the first model in a situation where lesion detection based on the second model is effective, thereby enabling accurate lesion detection.

[0072] (Modification 1-3) When it is determined that a predetermined condition based on the first score S1 is satisfied, the image processing device 1 may start the process of calculating the second score S2 and changing the threshold number of times Mth.

[0073] For example, after starting the lesion detection process, the image processing device 1 does not cause the second score calculation unit 33 to calculate the second score S2. If the image processing device 1 determines that the first score S1 exceeds the first score threshold Sth1, the image processing device 1 starts calculating the second score S2 using the second score calculation unit 33 and changes the threshold number of times Mth (or the first score threshold Sth1) in accordance with the second score S2, as in the above-described embodiment. On the other hand, if the image processing device 1 determines that the first score S1 has become equal to or less than the first score threshold Sth1 after starting calculation of the second score S2 using the second score calculation unit 33, the image processing device 1 again stops calculation of the second score S2 using the second score calculation unit 33. Note that the "predetermined condition" is not limited to the condition that the first score S1 is greater than the first score threshold Sth1, and may be any condition that determines that the likelihood of the presence of a lesion has increased. For example, examples of such conditions include a condition that the first score S1 is greater than a predetermined threshold value that is smaller than the first score threshold value Sth1, a condition that the increase in the first score S1 per unit time (i.e., the derivative of the first score S1) is greater than or equal to a predetermined value, and a condition that the number of consecutive times M exceeding the threshold value is greater than or equal to a predetermined value.

[0074] Furthermore, when a predetermined condition is satisfied and calculation of the second score S2 is started, the image processing device 1 may calculate the second score S2 going back to a past processing time and change the threshold number of times Mth (or the first score threshold Sth1) based on the second score S2. In this case, the image processing device 1 may store, for example, feature data calculated by the feature extraction unit 31 at a past processing time in the memory 12 or the like, and the second score calculation unit 33 may calculate the second score S2 at the past processing time based on the feature data and change the threshold number of times Mth (or the first score threshold Sth1) based on the second score S2.

[0075] According to this modification, the image processing device 1 can appropriately reduce the calculation load by limiting the period for calculating the second score S2.

[0076] (Modification 1-4) The image processing device 1 may process a video image made up of endoscopic images Ia generated during an endoscopic examination after the examination.

[0077] For example, at any time after an examination, when an image to be processed is specified based on user input via the input unit 14, the image processing device 1 repeatedly performs the processing of the flowchart shown in Figure 7 on the time-series endoscopic images Ia that constitute the image until it is determined that the target image has ended.

[0078] Second Embodiment (2-1) Overview In the second embodiment, the image processing device 1 performs lesion detection based on the second score S2 based on the second model, and changes the second score threshold Sth2, which is compared with the second score S2, based on the first score S1 based on the first model. This allows accurate detection of a lesion site in both situations where the first model is likely to accurately detect a lesion site and where the second model is likely to accurately detect a lesion site.

[0079] Hereinafter, the same components of the endoscopic examination system 100 as those in the first embodiment will be appropriately designated by the same reference numerals as those in the first embodiment, and their description will be omitted. The hardware configuration of the image processing device 1 according to the second embodiment is the same as the hardware configuration of the image processing device 1 shown in Fig. 2, and the functional block configuration of the processor 11 of the image processing device 1 according to the second embodiment is the same as the functional block configuration shown in Fig. 3.

[0080] In the second embodiment, during a period in which the number of consecutive exceeding-threshold values ​​M increases, the lesion detection unit 34 gradually or continuously lowers the second score threshold Sth2 (i.e., relaxes the conditions for determining that a lesion site has been detected) as the number of consecutive exceeding-threshold values ​​M increases. This allows the lesion detection unit 34 to appropriately relax the conditions for lesion detection based on the second model and accurately perform lesion detection even in a situation in which lesion detection based on the first model is effective.

[0081] In the second embodiment, the second model is an example of a “selection model,” and the first model is an example of a “non-selection model.” The second score threshold Sth2 is an example of a “parameter used for lesion detection based on the selection model.”

[0082] (2-2) Specific Examples Figure 8(A) is a graph showing the progress of the first score S1 from processing time t0 when acquisition of the endoscopic image Ia starts in the second embodiment, and Figure 8(B) is a graph showing the progress of the second score S2 from processing time t0 in the second embodiment. Note that the specific examples shown in Figures 8(A) and 8(B) are examples of lesion detection processing in a situation where the accuracy of lesion detection based on the first model is higher than the accuracy of lesion detection based on the second model.

[0083] In this case, at each processing time after processing time t0, the lesion detection unit 34 compares the first score S1 obtained at each processing time with the first score threshold Sth1, and the second score S2 with the second score threshold Sth2. Then, at processing time "t11", the lesion detection unit 34 determines that the first score S1 exceeds the first score threshold Sth1 and increases the number of consecutive exceeding-threshold values ​​M.

[0084] Then, after processing time t11, which is the start time of the period in which the consecutive exceed-threshold count M increases, the lesion detection unit 34 changes the second score threshold Sth2 in accordance with the consecutive exceed-threshold count M. Here, the lesion detection unit 34 continuously decreases the second score threshold Sth2 as the consecutive exceed-threshold count M increases. Then, at processing time t12, which is included in the period in which the consecutive exceed-threshold count M increases, the second score S2 becomes greater than the second score threshold Sth2, and the lesion detection unit 34 determines that a lesion area is present at processing time t12.

[0085] In this way, even in a situation where the accuracy of lesion detection based on the first model is higher than the accuracy of lesion detection based on the second model, the second score threshold Sth2 is decreased as the number of consecutive exceeding-threshold values ​​M increases, and the lesion detection conditions for the second score S2 based on the second model are suitably relaxed, thereby accurately performing lesion detection. Furthermore, even in a situation where the accuracy of lesion detection based on the second model is higher than the accuracy of lesion detection based on the first model, the second score S2 based on the second model reaches the second score threshold Sth2 even if the second score threshold Sth2 does not change, so the lesion detection unit 34 can accurately perform lesion detection.

[0086] (2-3) Processing Flow Fig. 9 is an example of a flowchart executed by the image processing device 1 in the second embodiment. The image processing device 1 repeatedly executes the processing of this flowchart until the end of the endoscopic examination.

[0087] First, the endoscopic image acquisition unit 30 of the image processing device 1 acquires an endoscopic image Ia (step S31). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope 3 via the interface 13. The display control unit 35 also executes processing such as displaying the endoscopic image Ia acquired in step S31 on the display device 2. The feature extraction unit 31 also generates feature data indicating the feature amounts of the acquired endoscopic image Ia.

[0088] Next, the second score calculation unit 33 calculates a second score S2 based on the variable number of endoscopic images Ia (step S32). In this case, for example, the second score calculation unit 33 calculates the second score S2 based on feature data of the variable number of endoscopic images Ia acquired at the current processing time and past processing times and a second model constructed based on the second model information storage unit D2. Furthermore, in parallel with step S12, the first score calculation unit 32 calculates a first score S1 based on a predetermined number of endoscopic images Ia (step S33). In this case, for example, the first score calculation unit 32 calculates the first score S1 based on feature data of the predetermined number of endoscopic images Ia acquired at the current processing time (and past processing times) and a first model constructed based on the first model information storage unit D1.

[0089] After executing step S33, the lesion detection unit 34 determines whether the first score S1 is greater than the first score threshold Sth1 (step S34). If the first score S1 is greater than the first score threshold Sth1 (step S34; Yes), the lesion detection unit 34 increments the number of consecutive over-threshold occurrences M by 1 (step S35). Note that the initial value of the number of consecutive over-threshold occurrences M is 0. On the other hand, if the first score S1 is equal to or less than the first score threshold Sth1 (step S34; No), the lesion detection unit 34 resets the number of consecutive over-threshold occurrences M to its initial value of 0 (step S36).

[0090] After execution of step S35 or step S36, the lesion detection unit 34 determines a second score threshold Sth2, which is a threshold to be compared with the second score S2, based on the number of consecutive occurrences exceeding the threshold value M (step S37). In this case, the lesion detection unit 34 refers to, for example, a pre-stored formula or lookup table, and reduces the second score threshold Sth2 as the number of consecutive occurrences exceeding the threshold value M increases.

[0091] After steps S32 and S37 are performed, the lesion detection unit 34 determines whether the second score S2 is greater than the second score threshold Sth2 (step S38). If the second score S2 is greater than the second score threshold Sth2 (step S38; Yes), the lesion detection unit 34 determines that a lesion is present and notifies the user that a lesion has been detected by at least one of displaying and / or outputting sound (step S39). On the other hand, if the second score S2 is equal to or less than the second score threshold Sth2 (step S38; No), the process returns to step S31.

[0092] (2-4) Modifications Next, a description will be given of modifications of the second embodiment. The following modifications may be combined in any manner.

[0093] (Variation 2-1) When it is determined that a predetermined condition based on the second score S2 is satisfied, the image processing device 1 may start the calculation of the first score S1 using the first model and the process of changing the second score threshold Sth2.

[0094] For example, after starting the lesion detection process, the image processing device 1 does not calculate the first score S1 using the first score calculation unit 32. If the second score S2 is greater than a predetermined threshold (e.g., 0) that is smaller than the second score threshold Sth2, the image processing device 1 starts calculating the first score S1 using the first score calculation unit 32 and changes the second score threshold Sth2 according to the number of consecutive exceeding-threshold values ​​M, as in the above-described embodiment. On the other hand, if the image processing device 1 determines that the second score S2 has become equal to or less than the predetermined threshold after starting calculation of the first score S1 using the first score calculation unit 32, the image processing device 1 again stops calculation of the first score S1 using the first score calculation unit 32. The "predetermined condition" is not limited to the condition that the second score S2 is greater than the predetermined threshold, but may be any condition that determines that the likelihood of the presence of a lesion has increased. For example, an example of such a condition includes a condition that the increase in the second score S2 per unit time (i.e., the derivative of the first score S1) is equal to or greater than a predetermined value.

[0095] Furthermore, when a predetermined condition is satisfied and calculation of the first score S1 is started, the image processing device 1 may calculate the first score S1 going back to a past processing time and change the second score threshold Sth2 based on the first score S1. In this case, the image processing device 1 may store, for example, feature data calculated by the feature extraction unit 31 at a past processing time in the memory 12, and the first score calculation unit 32 may calculate the first score S1 at the past processing time based on the feature data and change the second score threshold Sth2 at the past processing time based on the first score S1. In this case, the image processing device 1 compares the second score S2 with the second score threshold Sth2 at each past processing time to determine the presence or absence of a lesion.

[0096] According to this modification, the image processing device 1 can appropriately reduce the calculation load by limiting the period for calculating the second score S2.

[0097] (Modification 2-2) The image processing device 1 may process a video image made up of endoscopic images Ia generated during an endoscopic examination after the examination.

[0098] For example, at any time after an examination, when an image to be processed is specified based on user input via the input unit 14, the image processing device 1 repeatedly performs the processing of the flowchart shown in Figure 9 on the time-series endoscopic images Ia that constitute the image, until it is determined that the target image has ended.

[0099] Third Embodiment In a third embodiment, the image processing device 1 switches between lesion detection processing based on the first embodiment and lesion detection processing based on the second embodiment based on the degree of time-series fluctuation of the endoscopic image Ia. Hereinafter, the lesion detection processing based on the first embodiment will be referred to as "first model-based lesion detection processing," and the lesion detection processing based on the second embodiment will be referred to as "second model-based lesion detection processing."

[0100] Hereinafter, the same components of the endoscopic examination system 100 as those in the first embodiment will be appropriately designated by the same reference numerals as those in the first embodiment, and their description will be omitted. The hardware configuration of the image processing device 1 according to the third embodiment is the same as the hardware configuration of the image processing device 1 shown in Fig. 2, and the functional block configuration of the processor 11 of the image processing device 1 according to the third embodiment is the same as the functional block configuration shown in Fig. 3.

[0101] In the third embodiment, the lesion detection unit 34 calculates a score (also referred to as a "fluctuation score") representing the degree of fluctuation between an endoscopic image Ia at a time index t representing the current processing time (also referred to as a "currently processed image") and an endoscopic image Ia acquired immediately before that time (i.e., time index "t-1") (also referred to as a "previous image"). The greater the degree of fluctuation between the currently processed image and the previous image, the greater the value of the fluctuation score. For example, the lesion detection unit 34 calculates as the fluctuation score an arbitrary similarity index based on a comparison between images (i.e., a comparison between images). Examples of the similarity index in this case include a correlation coefficient, a structural similarity (SSIM) index, a peak signal-to-noise ratio (PSNR) index, and the squared error between corresponding pixels. In addition, instead of calculating the fluctuation score by directly comparing the currently processed image with the past image, the lesion detection unit 34 may compare the feature amounts of the currently processed image with the feature amounts of the past image and calculate the similarity between them as the fluctuation score.

[0102] Then, when the fluctuation score is equal to or less than a predetermined threshold (also referred to as the "fluctuation threshold"), the lesion detection unit 34 performs first-model-based lesion detection processing. That is, in this case, the lesion detection unit 34 determines a threshold number Mth based on the second score S2, and determines that a lesion exists when the number of consecutive occurrences exceeding the threshold M based on the first score S1 is greater than the threshold number Mth. The fluctuation threshold is pre-stored in, for example, the memory 12. On the other hand, when the fluctuation score is greater than the fluctuation threshold, the lesion detection unit 34 performs second-model-based lesion detection processing. That is, in this case, the lesion detection unit 34 determines a second score threshold Sth2 based on the first score S1, and determines that a lesion exists when the second score S2 is greater than the second score threshold Sth2. As described above, in the third embodiment, the lesion detection unit 34 selects a selected model, which is a model to be used for lesion detection, from the first model and the second model based on the degree of fluctuation of the endoscopic image Ia.

[0103] Here, a supplementary explanation of the effects of the third embodiment will be provided. As described in the first embodiment, lesion detection based on the first model has the advantage that lesion detection can be performed even under conditions in which the log-likelihood ratio based on the second model is unlikely to increase when there is no time change in the endoscopic image Ia (i.e., when the fluctuation score is relatively low). Lesion detection based on the second model has the advantage that it is resistant to instantaneous noise and can quickly detect easily identifiable lesion sites. Taking the above into consideration, in the third embodiment, the lesion detection unit 34 determines whether or not a lesion has been detected based on the first score S1 and the number of consecutive occurrences above the threshold M when the fluctuation score is equal to or less than the fluctuation threshold and lesion detection based on the first model is effective. Furthermore, when the fluctuation score is equal to or less than the fluctuation threshold and lesion detection based on the first model is effective, it determines whether or not a lesion has been detected based on the second score S2. This allows for favorable improvement in lesion detection accuracy.

[0104] FIG. 10 is an example of a flowchart executed by the image processing device 1 in the third embodiment.

[0105] First, the endoscopic image acquisition unit 30 of the image processing device 1 acquires the endoscopic image Ia (step S41). In this case, the endoscopic image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope 3 via the interface 13. In addition, the display control unit 35 executes processing such as displaying the endoscopic image Ia acquired in step S41 on the display device 2.

[0106] Next, the lesion detection unit 34 calculates a fluctuation score based on the current processing image, which is the endoscopic image Ia obtained in step S41 at the current processing time, and the previous image, which is the endoscopic image Ia obtained in step S41 at the immediately preceding processing time (step S42). The lesion detection unit 34 then determines whether the fluctuation score is greater than the fluctuation threshold (step S43). If the fluctuation score is greater than the fluctuation threshold (step S43; Yes), the image processing device 1 executes second model-based lesion detection processing (step S44). In this case, the image processing device 1 executes the flowchart of FIG. 9 , excluding the processing of step S31, which overlaps with step S41. Note that if it is determined in step S38 that the second score S2 is equal to or less than the second score threshold Sth2, the process may proceed to step S46. On the other hand, if the fluctuation score is equal to or less than the fluctuation threshold (step S43; No), the image processing device 1 executes first model-based lesion detection processing (step S45). In this case, the image processing device 1 executes the flowchart of FIG. 7 , excluding the processing of step S11, which overlaps with step S41. If it is determined in step S20 that the number of consecutive occurrences exceeding the threshold value M is equal to or less than the threshold value Mth, or if the processing of step S19 is completed, the processing may proceed to step S46.

[0107] The image processing device 1 then determines whether the endoscopic examination has ended (step S46). For example, the image processing device 1 determines that the endoscopic examination has ended when it detects a predetermined input to the input unit 14 or the operation unit 36. If the image processing device 1 determines that the endoscopic examination has ended (step S46; Yes), it ends the processing of the flowchart. On the other hand, if the image processing device 1 determines that the endoscopic examination has not ended (step S46; No), it returns the processing to step S41.

[0108] 11 is a block diagram of an image processing device 1X according to a fourth embodiment. The image processing device 1X includes an acquisition unit 30X and a lesion detection unit 34X. The image processing device 1X may be composed of multiple devices.

[0109] The acquisition means 30X acquires endoscopic images of a subject captured by an imaging unit provided in the endoscope. In this case, the acquisition means 30X may immediately acquire endoscopic images generated by the imaging unit, or may acquire endoscopic images generated in advance by the imaging unit and stored in a storage device at a predetermined timing. The acquisition means 30X may be, for example, the endoscopic image acquisition unit 30 in the first to third embodiments.

[0110] The lesion detection means 34X detects lesions based on a selected model selected from a first model that performs inference on lesions in a subject based on a predetermined number of endoscopic images and a second model that performs inference on lesions in a subject based on a variable number of endoscopic images. The lesion detection means 34X also changes parameters used for lesion detection based on the selected model, based on a non-selected model, which is the first model or the second model, that is not the selected model. The "selected model" may be the "first model" in the first-model-based lesion detection process in the first or third embodiment, or the "second model" in the second-model-based lesion detection process in the second or third embodiment. The "parameters used for lesion detection based on the selected model" may be the "threshold number Mth" or the "first score threshold Sth1" in the first-model-based lesion detection process in the first or third embodiment, or the "second score threshold Sth2" in the second-model-based lesion detection process in the second or third embodiment. The selection of the "selected model" and the "non-selected model" here is not limited to being made autonomously based on the fluctuation score as in the third embodiment, but may be predetermined by setting as in the first or second embodiment. The lesion detection means 34X may be, for example, the first score calculation unit 32, the second score calculation unit 33, and the lesion detection unit 34 in the first to third embodiments.

[0111] 12 is an example flowchart showing the processing procedure in the fourth embodiment. First, the acquisition means 30X acquires endoscopic images of a subject captured by an imaging unit provided in an endoscope (step S51). The lesion detection means 34X detects a lesion based on a selected model selected from a first model that performs inference regarding a lesion in the subject based on a predetermined number of endoscopic images and a second model that performs inference regarding a lesion in the subject based on a variable number of endoscopic images. Furthermore, the lesion detection means 34X changes parameters used for lesion detection based on the selected model based on either the first model that is not the selected model or a non-selected model that is the second model (step S52).

[0112] According to the fourth embodiment, the image processing device 1X can accurately detect a lesion site present in an endoscopic image.

[0113] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0114] In addition, some or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0115] [Supplementary Note 1] An image processing device comprising: an acquisition means for acquiring endoscopic images of a subject captured by an imaging unit provided in an endoscope; and a lesion detection means for detecting the lesion based on a selection model selected from a first model that makes an inference about a lesion in the subject based on a predetermined number of the endoscopic images and a second model that makes an inference about the lesion based on a variable number of the endoscopic images, wherein the lesion detection means changes a parameter used for detecting the lesion based on the selection model, based on a non-selection model that is the first model or the second model that is not the selection model. [Supplementary Note 2] The parameter is a parameter that specifies a condition for determining that the lesion has been detected, and the lesion detection means changes the parameter to relax the condition as the confidence that the lesion exists, indicated by a score calculated by the non-selection model, increases. [Supplementary Note 3] The image processing device according to Supplementary Note 1, wherein the first model is a deep learning model whose architecture includes a convolutional neural network. [Supplementary Note 4] The image processing device according to Supplementary Note 1, wherein the selection model is the first model, and the lesion detection means determines that the lesion has been detected when the number of consecutive times that the certainty of the existence of the lesion indicated by the score calculated by the first model from the endoscopic images acquired in time series becomes greater than a predetermined threshold is greater than a predetermined number of times, the parameter is at least one of the predetermined number of times or the predetermined threshold, and the lesion detection means changes at least one of the predetermined number of times or the predetermined threshold based on the score calculated by the second model. [Supplementary Note 5] The image processing device according to Supplementary Note 1, wherein the second model is a model based on SPRT. [Supplementary Note 6] The image processing device according to Supplementary Note 1, wherein the selection model is the second model, and the lesion detection means determines that the lesion has been detected when the certainty of the existence of the lesion indicated by the score calculated by the second model becomes greater than a predetermined threshold, the parameter is the predetermined threshold, and the lesion detection means changes the predetermined threshold based on the score calculated by the first model.[Supplementary Note 7] The image processing device of Supplementary Note 1, wherein the lesion detection means determines the selected model from the first model and the second model based on the degree of variation in the endoscopic image. [Supplementary Note 8] The image processing device of Supplementary Note 1, wherein the lesion detection means starts calculating a score using the non-selected model when it is determined that a predetermined condition based on the score calculated by the selected model is satisfied. [Supplementary Note 9] The image processing device of Supplementary Note 1, further comprising output control means for displaying or outputting audio information related to the lesion detection result by the lesion detection means. [Supplementary Note 10] The image processing device of Supplementary Note 9, wherein the output control means outputs information related to the lesion detection result and information related to the selected model to support the examiner in making a decision. [Supplementary Note 11] An image processing method, in which a computer acquires endoscopic images of a subject captured by an imaging unit provided in an endoscope, detects the lesion based on a selected model selected from a first model that makes an inference about a lesion in the subject based on a predetermined number of the endoscopic images and a second model that makes an inference about the lesion based on a variable number of the endoscopic images, and changes parameters used for detecting the lesion based on the selected model based on a non-selected model that is the first model or second model that is not the selected model. [Supplementary Note 12] A storage medium storing a program that causes a computer to execute the following processes: acquires endoscopic images of a subject captured by an imaging unit provided in an endoscope, detects the lesion based on a selected model selected from a first model that makes an inference about a lesion in the subject based on a predetermined number of the endoscopic images and a second model that makes an inference about the lesion based on a variable number of the endoscopic images, and changes parameters used for detecting the lesion based on the selected model based on a non-selected model that is the first model or second model that is not the selected model.

[0116] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0117] REFERENCE SIGNS LIST 1, 1X image processing device 2 display device 3 endoscope 11 processor 12 memory 13 interface 14 input unit 15 light source unit 16 sound output unit 100 endoscopic examination system

Claims

1. an acquisition means for acquiring an endoscopic image of a subject by an imaging unit provided in the endoscope; a lesion detection means for detecting the lesion based on a selection model selected from a first model for making an inference regarding a lesion in the subject based on a predetermined number of the endoscopic images and a second model for making an inference regarding the lesion based on a variable number of the endoscopic images; having The lesion detection means is an image processing device that changes parameters used for detecting the lesion based on the selected model, based on a non-selected model that is a first model or a second model other than the selected model.

2. the parameter defines a condition for determining that the lesion has been detected, The image processing device according to claim 1 , wherein the lesion detection means changes the parameters so as to relax the conditions as the degree of certainty of the presence of the lesion indicated by the score calculated by the non-selection model becomes higher.

3. The image processing device according to claim 1 , wherein the first model is a deep learning model including a convolutional neural network in its architecture.

4. the selected model is the first model, the lesion detection means determines that the lesion has been detected when a consecutive number of times that a certainty of the presence of the lesion, indicated by a score calculated by the first model from the endoscopic images acquired in time series, becomes greater than a predetermined threshold value is greater than a predetermined number of times; the parameter is at least one of the predetermined number of times or the predetermined threshold value, The image processing device according to claim 1 , wherein the lesion detection means changes at least one of the predetermined number of times or the predetermined threshold value based on the score calculated by the second model.

5. The image processing device according to claim 1 , wherein the second model is a model based on SPRT.

6. the selected model is the second model, the lesion detection means determines that the lesion has been detected when a certainty that the lesion exists, which is indicated by a score calculated by the second model, is greater than a predetermined threshold; the parameter is the predetermined threshold, The image processing device according to claim 1 , wherein the lesion detection means changes the predetermined threshold value based on the score calculated by the first model.

7. The image processing apparatus according to claim 1 , wherein the lesion detection means determines the selected model from the first model and the second model based on a degree of variation in the endoscopic image.

8. The image processing device according to claim 1 , wherein the lesion detection means starts calculation of the score using the non-selected model when it is determined that a predetermined condition based on the score calculated by the selected model is satisfied.

9. The image processing apparatus according to claim 1 , further comprising an output control unit that displays or outputs audio information relating to the result of the lesion detection by the lesion detection unit.

10. The image processing apparatus according to claim 9 , wherein the output control means outputs information about the lesion detection result and information about the selection model to assist an examiner in making a decision.

11. The computer An endoscopic image of the subject is obtained by an imaging unit provided in the endoscope; Detecting the lesion based on a selection model selected from a first model that performs inference regarding a lesion in the subject based on a predetermined number of the endoscopic images and a second model that performs inference regarding the lesion based on a variable number of the endoscopic images; changing a parameter used for detecting the lesion based on the selected model based on a non-selected model which is a first model or a second model other than the selected model; Image processing methods.

12. An endoscopic image of the subject is obtained by an imaging unit provided in the endoscope; Detecting the lesion based on a selection model selected from a first model that performs inference regarding a lesion in the subject based on a predetermined number of the endoscopic images and a second model that performs inference regarding the lesion based on a variable number of the endoscopic images; A program that causes a computer to execute a process of changing parameters used for detecting the lesion based on the selected model, based on a non-selected model that is a first model or a second model that is not the selected model.