Image processing device, image processing method and program

Abstract

This image processing device 1X comprises an acquisition means 30X, a score calculation means 31X, and a classification means 32X. The acquisition means 30X acquires an endoscopic image obtained by imaging a subject using an imaging unit provided to an endoscope. The score calculation means 31X calculates a score relating to the likelihood of each candidate class corresponding to a type of lesion, the candidate classes serving as candidates for classifying the image group of the acquired endoscopic image. The classification means 32X classifies the image group when at least one of the scores is assessed to have reached a threshold value.

Description

Image processing device, image processing method, and storage medium

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a storage medium that process images acquired during an endoscopic examination.

[0002] Conventionally, endoscopic systems that display images of the inside of organ lumens have been known. For example, Patent Literature 1 discloses a learning method for a learning model that outputs information about a lesion site contained in endoscopic image data when the endoscopic image data generated by an imaging device is input. Furthermore, Patent Literature 2 discloses a classification method for classifying sequential data using a technique that applies a sequential probability ratio test (SPRT). Furthermore, Non-Patent Literature 1 discloses a matrix approximation method for multi-class classification in the SPRT-based technique disclosed in Patent Literature 2.

[0003] International Publication WO2020 / 003607 International Publication WO2020 / 194497

[0004] Miyagawa Taiki, and Akinori F. Ebihara. "The Power of Log-Sum-Exp: Sequential Density Ratio Matrix Estimation for Speed-Accuracy Optimization." International Conference on Machine Learning. PMLR, 2021.

[0005] When classifying lesions, a qualitative diagnosis, from images taken during endoscopic examinations, methods that analyze each image individually can result in inaccurate classification of lesions that are difficult to distinguish from a single image. Furthermore, when classifying lesions from multiple images, it can be difficult to set an appropriate number of images to use.

[0006] In view of the above-mentioned problems, one object of the present disclosure is to provide an image processing device, an image processing method, and a storage medium that are capable of appropriately classifying lesion sites in endoscopic images.

[0007] One aspect of the image processing device is an image processing device having: an acquisition means for acquiring endoscopic images of a subject captured by an imaging unit provided in an endoscope; a score calculation means for calculating a score relating to the likelihood of each candidate class corresponding to a type of lesion into which a group of acquired endoscopic images are to be classified; and a classification means for classifying the group of images when it is determined that at least one of the scores has reached a threshold value.

[0008] One aspect of the image processing method is an image processing method in which a computer acquires endoscopic images of a subject taken by an imaging unit provided in an endoscope, calculates a score relating to the likelihood of each candidate class corresponding to a type of lesion into which a group of acquired endoscopic images are to be classified, and classifies the group of images when it is determined that at least one of the scores has reached a threshold value.

[0009] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute a process of acquiring endoscopic images of a subject using an imaging unit provided in an endoscope, calculating a likelihood score for each candidate class corresponding to a type of lesion into which a group of acquired endoscopic images are to be classified, and classifying the group of images when it is determined that at least one of the scores has reached a threshold value.

[0010] One example of the effect of the present disclosure is that it becomes possible to appropriately classify lesion sites in endoscopic images.

[0011] 1 shows a schematic configuration of an endoscopic examination system; FIG. 2 shows a hardware configuration of an image processing device; FIG. 3 is a functional block diagram of an image processing device; FIG. 4 is a graph showing the progress of classification scores; FIG. 5 shows a first display example of a display screen displayed by a display device during endoscopic examination; FIG. 6 shows a second display example of a display screen displayed by a display device during endoscopic examination; FIG. 7 shows a third display example of a display screen displayed by a display device during endoscopic examination; FIG. 8 is an example of a flowchart executed by an image processing device; FIG. 9 is a block diagram of an image processing device in a second embodiment; FIG. 10 is an example of a flowchart executed by an image processing device in the second embodiment.

[0012] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium will be described with reference to the drawings.

[0013] <First Embodiment> (1) System Configuration Fig. 1 shows a schematic configuration of an endoscopic examination system 100. The endoscopic examination system 100 performs qualitative diagnostic classification of a region of a subject suspected of having a lesion (lesion region) for an examiner such as a doctor who performs an examination or treatment using an endoscope, and presents the classification results. As shown in Fig. 1 , the endoscopic examination system 100 mainly includes an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1.

[0014] The image processing device 1 acquires images (also referred to as "endoscopic images Ia") captured by the endoscope 3 in a time series from the endoscope 3, and displays a screen based on the endoscopic images Ia on the display device 2. The endoscopic images Ia are images captured at a predetermined frame rate during at least one of the steps of inserting or ejecting the endoscope 3 into the subject. In this embodiment, when the image processing device 1 detects an endoscopic image Ia (also referred to as a "lesion image") that includes a lesion site, it classifies the lesion site based on the time series lesion images and displays information related to the classification result on the display device 2.

[0015] The display device 2 is a display or the like that displays a predetermined image based on a display signal supplied from the image processing device 1 .

[0016] The endoscope 3 mainly comprises an operation unit 36 ​​through which the examiner makes predetermined inputs, a flexible shaft 37 that is inserted into the subject's organ to be imaged, a tip 38 that incorporates an imaging unit such as a micro-imaging element, and a connection unit 39 for connecting to the image processing device 1. In this embodiment, the operation unit 36 ​​includes a button (also referred to as a "still image save button") that instructs the examiner to capture (i.e., save as a still image) the endoscopic image displayed on the display device 2 when the examiner determines that an endoscopic image including a tumor site has been displayed on the display device 2.

[0017] 1 is an example, and various modifications may be made. For example, the image processing device 1 may be configured integrally with the display device 2. In another example, the image processing device 1 may be configured from multiple devices.

[0018] Hereinafter, processing in an endoscopic examination of the large intestine will be described as a representative example, but the subject is not limited to the large intestine, and the esophagus or stomach may also be the subject of the examination. Examples of endoscopes that are subject to the present disclosure include pharyngeal endoscopes, bronchoscopes, upper gastrointestinal endoscopes, duodenoscopes, small intestinal endoscopes, colonoscopes, capsule endoscopes, thoracoscopes, laparoscopes, cystoscopes, cholangioscopes, arthroscopes, spinal endoscopes, angioscopes, and epidural endoscopes. Examples of pathological conditions at lesion sites that are subject to detection in the present disclosure include (a) to (f) below. (a) Head and neck: pharyngeal cancer, malignant lymphoma, papilloma (b) Esophagus: esophageal cancer, esophagitis, hiatal hernia, Barrett's esophagus, esophageal varices, esophageal achalasia, esophageal submucosal tumor, benign esophageal tumor (c) Stomach: gastric cancer, gastritis, gastric ulcer, gastric polyp, gastric tumor (d) Duodenum: duodenal cancer, duodenal ulcer, duodenitis, duodenal tumor, duodenal lymphoma (e) Small intestine: small intestine cancer, small intestine neoplastic disease, small intestine inflammatory disease, small intestine vascular disease (f) Large intestine: large intestine cancer, large intestine neoplastic disease, large intestine inflammatory disease, large intestine polyp, large intestine polyposis, Crohn's disease, colitis, intestinal tuberculosis, hemorrhoids

[0019] (2) Hardware Configuration Fig. 2 shows the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, and a sound output unit 16. These elements are connected via a data bus 19.

[0020] The processor 11 performs predetermined processing by executing programs stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0021] The memory 12 is composed of various volatile memories used as working memories, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memories that store information necessary for processing by the image processing device 1. The memory 12 may include an external storage device such as a hard disk connected to or built into the image processing device 1, or may include a storage medium such as a removable flash memory. The memory 12 stores programs for the image processing device 1 to execute each process in this embodiment.

[0022] The memory 12 also functionally includes a first calculation information storage unit D1 that stores the first calculation information and a second calculation information storage unit D2 that stores the second calculation information. The first calculation information and the second calculation information are information used by the image processing device 1 to classify a lesion site or information indicating the calculation results in the classification process, and will be described in detail below. The memory 12 also stores various parameters necessary for calculating scores for classifying lesion sites. Note that at least a portion of the information stored in the memory 12 may be stored by an external device other than the image processing device 1. In this case, the external device may be one or more server devices that can communicate data with the image processing device 1 via a communication network or by direct communication.

[0023] The interface 13 acts as an interface between the image processing device 1 and an external device. For example, the interface 13 supplies the display information "Ib" generated by the processor 11 to the display device 2. The interface 13 also supplies light generated by the light source unit 15 to the endoscope 3. The interface 13 also supplies an electrical signal indicating the endoscopic image Ia supplied from the endoscope 3 to the processor 11. The interface 13 may be a communication interface such as a network adapter for wired or wireless communication with an external device, or may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like.

[0024] The input unit 14 generates an input signal based on an operation by the examiner. The input unit 14 is, for example, a button, a touch panel, a remote controller, or a voice input device. The light source unit 15 generates light to be supplied to the tip 38 of the endoscope 3. The light source unit 15 may also incorporate a pump or the like for sending water or air to be supplied to the endoscope 3. The sound output unit 16 outputs sound based on the control of the processor 11.

[0025] (3) Overview of Lesion Site Detection Processing Next, we will explain an overview of the lesion site detection processing performed by the image processing device 1. In general, the image processing device 1 classifies lesion sites based on a variable number of time-series lesion images. As a result, the image processing device 1 accurately classifies lesion sites, which is difficult to do with a single image, and presents the classification results.

[0026] Fig. 3 is a functional block diagram of the image processing device 1. As shown in Fig. 3, the processor 11 of the image processing device 1 functionally includes a lesion image acquisition unit 30, a score calculation unit 31, a classification unit 32, and a display control unit 33. Note that in Fig. 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown in Fig. 3. The same applies to other functional block diagrams described later.

[0027] The lesion image acquisition unit 30 acquires endoscopic images Ia captured by the endoscope 3 via the interface 13 at predetermined intervals in accordance with the frame period of the endoscope 3, and selects a lesion image from the acquired endoscopic images Ia. The lesion image acquisition unit 30 then supplies the selected lesion image to the score calculation unit 31 and the display control unit 33. The lesion image acquisition unit 30 also supplies the acquired endoscopic image Ia to the display control unit 33.

[0028] Here, a specific example of a method for selecting a lesion image will be described. For example, when the lesion image acquisition unit 30 detects that the still image save button has been selected based on a signal supplied from the operation unit 36 ​​(i.e., an external input based on a user operation), it acquires, as the lesion image, the endoscopic image Ia displayed on the display device 2 at the time of selection. In this case, the lesion image acquisition unit 30 may acquire, as the lesion image, the most recent endoscopic image Ia received from the endoscope 3 at the time the still image save button was selected.

[0029] In another example, the lesion image acquisition unit 30 may select a lesion image without relying on an operation by an examiner (i.e., external input). For example, the lesion image acquisition unit 30 may acquire a lesion image based on a model for detecting lesion images (also referred to as a "lesion detection model"). In this case, parameters of the lesion detection model are pre-stored in the memory 12, etc., and the lesion image acquisition unit 30 inputs the endoscopic image Ia supplied from the endoscope 3 into the lesion detection model configured with reference to the above parameters, and determines whether the input endoscopic image Ia is a lesion image based on information output by the lesion detection model when the endoscopic image Ia is input. In this case, the lesion detection model is, for example, a classification model trained to, when the endoscopic image Ia is input, output a classification result regarding the presence or absence of a lesion site in the input endoscopic image Ia. For example, if the lesion detection model is configured using a neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and filter size of each filter in each layer, and the weight of each element of each filter are pre-stored in the memory 12, etc.

[0030] The display control unit 33, which will be described later, may assist the examiner in pressing the still image save button by displaying the lesion detection result based on the lesion detection model together with the latest endoscopic image Ia. For example, when the lesion detection model detects a lesion site, the display control unit 33 may highlight the latest endoscopic image Ia displayed on the display device 2 using a border effect or the like to prompt the examiner to press the still image save button.

[0031] The score calculation unit 31, classification unit 32, and display control unit 33 perform the processing described below using a period that is the time interval at which the lesion image acquisition unit 30 acquires lesion images. Hereinafter, the timing of processing based on this period will also be referred to as the "processing time."

[0032] The score calculation unit 31 is used for classification of the class to which the lesion site belongs, and calculates a score (also referred to as a "classification score") indicating the likelihood that the lesion site belongs to each candidate class (also referred to as a "candidate class") to which the lesion site may belong. In this case, the score calculation unit 31 calculates the classification score for each candidate class using lesion images acquired in time series by a method based on SPRT described in Patent Document 2 and Non-Patent Document 1. The number of candidate classes and the type of lesion site corresponding to each candidate class are set in advance for each subject.

[0033] The score calculation unit 31 functionally includes a first calculation unit 311 and a second calculation unit 312 .

[0034] The first calculation unit 311 calculates likelihood ratios for the latest "N" (N is an integer) lesion images at each processing time and supplies the calculation results to the second calculation unit 312. The "likelihood ratio" is an index indicating the likelihood of the class to which the lesion image belongs. The likelihood ratio increases when the correct class is in the numerator of the likelihood ratio, and decreases when the correct class is in the denominator of the likelihood ratio. In this case, for example, when N lesion images are input, the first calculation unit 311 calculates the likelihood ratios using a likelihood ratio calculation model trained to output likelihood ratios for the input N lesion images. The likelihood ratio calculation model may be a deep learning model, any other machine learning model, or a statistical model. In this case, for example, trained parameters of the likelihood ratio calculation model are stored in the memory 12, and the first calculation unit 311 inputs the latest N lesion images into a likelihood ratio calculation model configured by referring to the parameters and obtains the likelihood ratios output by the model. When the likelihood ratio calculation model is configured using a neural network, various parameters such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter are stored in advance in the memory 12. Note that even when the number of acquired lesion images is less than N, the first calculator 311 can acquire likelihood ratios from less than N lesion images using the likelihood ratio calculation model.

[0035] Here, the likelihood ratio calculation model may include an optional feature extractor that extracts feature quantities (i.e., feature vectors) of each lesion image input to the likelihood ratio calculation model, or may be configured separately from such a feature extractor. In the latter case, the likelihood ratio calculation model is a model trained to output likelihood ratios for each candidate class related to the N lesion images when the feature quantities of the N lesion images from which the feature quantities have been extracted by the feature extractor are input. Furthermore, the feature extractor may preferably extract feature quantities representing relationships between time-series data based on an optional method for calculating relationships between time-series data, such as LSTM (Long Short Term Memory).

[0036] Preferably, the first calculation unit 311 sets the number of images N according to the type of subject. For example, if the subject is an organ (e.g., a stomach) that allows the endoscope to be moved to a certain extent, the correlation between lesion images will be relatively low, and therefore the first calculation unit 311 sets the number of images N to a smaller value than for other subjects. On the other hand, if the subject is an organ (e.g., an esophagus) that does not allow the endoscope to be moved very much, the correlation between lesion images will be relatively high, and therefore the first calculation unit 311 sets the number of images N to a larger value than for other subjects. This allows the first calculation unit 311 to calculate the likelihood ratio more accurately. Similarly, a likelihood ratio calculation model may be prepared for each type of subject. In this case, a likelihood ratio calculation model is trained for each type of subject, and parameters obtained by the training are pre-stored in the memory 12 or the like for each type of subject. Note that the image processing device 1 may recognize the type of subject based on an external input from the input unit 14 before the endoscopic examination, or may automatically recognize the type of subject by applying any image recognition technology to and analyzing the endoscopic image Ia obtained at the start of the endoscopic examination.

[0037] The first calculation unit 311 stores the calculated likelihood ratio and the data used to calculate the likelihood ratio as first calculation information in the first calculation information storage unit D1. The "data used to calculate the likelihood ratio" may be the lesion image used to calculate the likelihood ratio, or may be feature amounts extracted from the lesion image.

[0038] The second calculation unit 312 calculates a likelihood ratio (also referred to as an "integrated likelihood ratio") by integrating the likelihood ratios calculated in time series, and determines a classification score based on the integrated likelihood ratio. Note that the classification score may be the integrated likelihood ratio itself, or may be a function that includes the integrated likelihood ratio as a variable.

[0039] Here, for the sake of simplicity, a specific method for calculating an integrated likelihood ratio when performing two-class classification will be described first.

[0040] The time when the first lesion image was obtained is defined as time index "1," and the current processing time is defined as time index "t." Any lesion image to be processed or its feature amount is defined as "x ｉ(i=1, ..., t). Here, the time index is incremented by 1 each time a lesion image is obtained. The t lesion images to be processed are an example of an "image group."

[0041] In this case, the candidate class "C ０ " and candidate class "C １ When " exists, the combined likelihood ratio of the candidate class C1 is expressed by the following equation (1).

[0042]

[0043] Here, "p" represents the probability of belonging to each candidate class (i.e., the confidence level between 0 and 1). In calculating the term on the right side of equation (1), the first calculation unit 311 can use the likelihood ratio stored in the first calculation information storage unit D1 as the first calculation information. ０ The integrated likelihood ratio is the inverse of equation (1).

[0044] In Equation (1), the time index t, which represents the current processing time, increases over time, so the length of the time series of lesion images (or their feature quantities) used to calculate the integrated likelihood ratio is variable. Thus, by using the integrated likelihood ratio based on Equation (1), the second calculation unit 312 can calculate the classification score taking into account a variable number of lesion images. Additionally, by using the integrated likelihood ratio based on Equation (1), a second advantage is that time-dependent features can be classified, and a third advantage is that a classification score can be suitably calculated that allows for classification with little loss of accuracy even in difficult-to-distinguish data.

[0045] Next, calculation of the integrated likelihood ratio of each candidate class when performing classification into three or more classes (multi-class classification) will be described. Assuming the number of candidate classes is "M" (M is an integer equal to or greater than 3), the score calculation unit 31 calculates the integrated likelihood ratio between the kth (k=1, 2, ..., M) candidate class among the M candidate classes and all candidate classes other than the kth candidate class. In this case, the score calculation unit 31 calculates the integrated likelihood ratio by, for example, replacing the denominators of the first and second terms on the right side of Equation (1) with the maximum likelihood of all candidate classes other than the kth candidate class. Note that in this case, the score calculation unit 31 may calculate the integrated likelihood ratio using the sum of the likelihoods of all candidate classes other than the kth candidate class instead of the maximum likelihood. Therefore, for example, when N lesion images from which features have been extracted by the feature extractor or the features thereof are input to the likelihood ratio calculation model, the score calculation unit 31 calculates the integrated likelihood ratio of each candidate class based on the likelihood ratio of each candidate class output by the likelihood ratio calculation model (i.e., the likelihood ratio shown on the right side of Equation (1)). The integrated likelihood ratio and classification score may be calculated by the methods described in Patent Document 2 and Non-Patent Document 1, instead of the above method.

[0046] The second calculation unit 312 stores the integrated likelihood ratio and classification score of each candidate class calculated at each processing time when a lesion image was obtained as second calculation information in the second calculation information storage unit D2.

[0047] The classification unit 32 classifies the lesion site based on the classification score calculated by the second calculation unit 312, and supplies the classification result to the display control unit 33. In this case, the classification unit 32 compares the classification score of each candidate class of the lesion site with a predetermined threshold (also referred to as "threshold Th"), and determines whether or not there is a candidate class whose classification score is equal to or greater than the threshold Th.

[0048] If a candidate class exists whose classification score is equal to or greater than the threshold value Th, the classification unit 32 outputs the candidate class whose classification score is equal to or greater than the threshold value Th as the classification result for the lesion site appearing in the group of lesion images used to calculate the classification score. Here, a candidate class with a higher classification score is more likely to be the class to which the lesion site appearing in the group of lesion images used to calculate the classification score belongs. The threshold value Th is, for example, a compatibility value determined based on experiments or the like, and is stored in advance in the memory 12 or the like. Thereafter, the classification unit 32 supplies a notification to the score calculation unit 31 to reset the classification score calculation process (i.e., update the start time).

[0049] On the other hand, if the classification unit 32 determines that there is no candidate class whose classification score is greater than or equal to the threshold value Th, it instructs the score calculation unit 31 to calculate a classification score for the group of lesion images including the lesion images acquired by the lesion image acquisition unit 30 after the determination.

[0050] Note that even if the classifier 32 determines that there is no candidate class whose classification score is equal to or greater than the threshold value Th, the classifier 32 may determine a classification if a predetermined condition other than the condition based on the threshold value Th is satisfied. For example, the classifier 32 may determine a classification if the time index t representing the current processing time is equal to or greater than a predetermined threshold value (i.e., if the number of images in the group of lesion images used is equal to or greater than a predetermined number). In this case, the classifier 32 outputs the candidate class with the highest classification score at the time when the time index t representing the current processing time is equal to or greater than the predetermined threshold value as the classification result for the lesion site represented in the lesion image used to calculate the classification score. The set value of the above-mentioned predetermined threshold value (predetermined number of images) is stored in advance in the memory 12, for example.

[0051] Furthermore, the classification unit 32 may determine that the classification score should be reset when it determines that no candidate class exists whose classification score is equal to or greater than the threshold value Th and when the time index t representing the current processing time is equal to or greater than a predetermined threshold value (i.e., when the number of lesion images used is equal to or greater than a predetermined number). In this case, the classification unit 32 instructs the score calculation unit 31 to reset the classification score calculation process (i.e., update the start time) without determining the classification. In this case, the first calculation unit 311 and the second calculation unit 312 of the score calculation unit 31 reset the classification score (as well as the first calculation information and the second calculation information) of each candidate class and calculate the classification score based on the group of lesion images newly acquired from the time the reset was performed. The set value of the above-mentioned predetermined threshold value (predetermined number) is stored in advance in the memory 12, for example.

[0052] The display control unit 33 generates display information Ib based on the endoscopic image Ia (including the lesion image) and the classification result supplied from the classification unit 32, and supplies the display information Ib to the display device 2 via the interface 13, thereby causing information related to the endoscopic image Ia and the classification result by the classification unit 32 to be displayed on the display device 2. Preferably, the display control unit 33 may also cause the display device 2 to display information related to the classification score stored in the second calculation information storage unit D2, etc. Display examples of the display control unit 33 will be described later.

[0053] Here, each component of the lesion image acquisition unit 30, score calculation unit 31, classification unit 32, and display control unit 33 can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording the necessary program on an arbitrary non-volatile storage medium and installing it as needed. Note that at least a portion of each component is not limited to being realized by software programs, but may also be realized by any combination of hardware, firmware, and software. Furthermore, at least a portion of each component may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program composed of each of the above components. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0054] (4) Example of Calculation of Classification Score Next, an example of calculation of the classification score will be described. FIG. 4 is a graph showing the progression of the classification score. In this example, the image processing device 1 starts processing at time "t0" and calculates the classification score for each of three candidate classes ("adenoma," "hyperplastic polyp," and "invasive carcinoma") based on the lesion images (lesion image A to lesion image D) acquired by the lesion image acquisition unit 30 at times "t1," "t2," "t3," and "t4," respectively. Here, graph G1 shows the progression of the classification score for the candidate class "adenoma," graph G2 shows the progression of the classification score for the candidate class "hyperplastic polyp," and graph G3 shows the progression of the classification score for the candidate class "invasive carcinoma."

[0055] First, at time t1, image processing device 1 calculates a classification score for each candidate class based on lesion image A obtained at time t1. Then, at time t2, image processing device 1 calculates a classification score for each candidate class based on lesion image B obtained at time t2 and lesion image A obtained at time t1. Then, at time t3, image processing device 1 calculates a classification score for each candidate class based on lesion image C obtained at time t3 and lesion images A and B obtained previously, and then, at time t4, calculates a classification score for each candidate class based on lesion image D obtained at time t4 and lesion images A to C obtained previously.

[0056] Then, as shown in graph G1, at time t4, the classification score of the candidate class "adenoma" becomes equal to or greater than the threshold value Th, and therefore the image processing device 1 generates a classification result that the lesion areas shown in lesion images A to D obtained from time t0 to time t4 are "adenoma."

[0057] In this way, the image processing device 1 sequentially calculates classification scores for image groups containing one or more input images, and performs classification when the classification score reaches a threshold. This allows for optimal improvement of classification performance by using an image group containing an optimal number of images for classification, without excessive or insufficient images. On the other hand, when lesion classification is performed using a fixed number of image groups, the quality of each image used is not taken into consideration, making it difficult to fix the optimal number of images to be used for classification. For example, using a large number of images can result in images containing noise such as blurring being introduced as the number of images used increases. Using a small number of images results in classification with low classification confidence, increasing the likelihood of misclassification in either case. Taking the above into consideration, the image processing device 1 according to this embodiment uses a variable number of image groups and performs classification when the classification score reaches a threshold, thereby optimally improving classification performance by using an optimal number of image groups for classification, without excessive or insufficient images.

[0058] (5) Display Example Next, the display control of the display device 2 executed by the display control unit 33 will be described.

[0059] 5 shows a first display example of a display screen displayed by the display device 2 during an endoscopic examination. The display control unit 33 of the image processing device 1 outputs to the display device 2 display information Ib generated based on the endoscopic image Ia and the lesion image acquired by the lesion image acquisition unit 30 and the classification results generated by the classification unit 32. The display control unit 33 transmits the endoscopic image Ia and the display information Ib to the display device 2, thereby causing the display device 2 to display the above-mentioned display screen. In this example, the lesion image acquisition unit 30 acquires a still image specified by the examiner operating the operation unit 36 ​​as the lesion image.

[0060] In the first display example, the display control unit 33 of the image processing device 1 provides a real-time image display area 70, a latest still image display area 71, a classification result display area 72, and a score transition display area 73 on the display screen.

[0061] Here, the display control unit 33 displays a moving image representing the latest endoscopic image Ia in the real-time image display area 70. The display control unit 33 also displays the latest still image (i.e., the latest lesion image acquired by the lesion image acquisition unit 30) in the latest still image display area 71.

[0062] Furthermore, the display control unit 33 displays the classification results by the classification unit 32 in the classification result display area 72. Note that, at the time of displaying the display screen shown in Fig. 5, the classification results of none of the candidate classes have reached the threshold value Th, and therefore the display control unit 33 displays a text message in the classification result display area 72 indicating that analysis is ongoing and prompting the user to continue specifying a still image (i.e., a lesion image).

[0063] In addition, in the score transition display area 73, the display control unit 33 displays a score transition graph (here, a diagram corresponding to FIG. 4 ) showing the transition of the classification score for each candidate class from the start of the endoscopic examination to the present time. In this case, the display control unit 33 also displays, in the score transition graph, still images (i.e., lesion images) used to calculate the classification scores in association with the time at which the still images were specified. This allows the display control unit 33 to present to the examiner the relationship between the obtained still images and changes in the classification scores. The score transition graph displayed in the score transition display area 73 is an example of a "diagram showing score transition."

[0064] Thus, in the first display example, when the classification is undetermined, the display control unit 33 outputs information indicating that the classification is undetermined, together with information regarding the classification score, etc. This allows the display control unit 33 to preferably visualize the current status of the classification process for the lesion site.

[0065] Furthermore, instead of or in addition to the display control shown in the first display example, when the classification is undetermined, the display control unit 33 may instruct the sound output unit 16 to output voice guidance or a predetermined warning sound to notify the examiner that the classification is undetermined. This also allows the display control unit 33 to preferably let the examiner know that the classification is undetermined.

[0066] 6 shows a second display example of the display screen displayed by the display device 2 during an endoscopic examination. In the second display example, the classifier 32 determines that the classification score of the candidate class "adenoma" supplied from the score calculator 31 is equal to or greater than the threshold value Th, and supplies a classification result indicating classification into the candidate class "adenoma" to the display controller 33. In this case, the display controller 33 displays a text message in the classification result display area 72, based on the above-mentioned classification result, indicating that there is a high possibility that an adenoma is present.

[0067] As described above, in the second display example, when the classification is determined, the display control unit 33 outputs information indicating the classification result (here, a text message in the classification result display area 72). This allows the display control unit 33 to appropriately notify the examiner of the classification result of the lesion site. Note that, instead of or in addition to the display control shown in the second display example, the display control unit 33 may instruct the sound output unit 16 to output voice guidance or a predetermined warning sound to notify the examiner of the classification result. This also allows the display control unit 33 to allow the examiner to understand the classification result.

[0068] 7 shows a third display example of the display screen displayed by the display device 2 during an endoscopic examination. In the third display example, the display control unit 33 enlarges and displays, in the score transition display area 73, a still image (i.e., a lesion image) and classification score obtained at a time specified by the examiner.

[0069] Specifically, the display control unit 33 selectably displays objects 74 (74A to 74D) associated with the respective times at which still images were obtained in the score transition display area 73. When the display control unit 33 detects that one of the objects 74 has been selected, it displays the still image (i.e., the lesion image) and classification scores corresponding to the selected object 74 in a balloon window 75. In this example, the display control unit 33 detects that object 74C has been selected, and displays a still image 76 and the classification scores of each candidate class at the time corresponding to object 74C in the balloon window 75. This makes it possible to present the still images and classification scores at any time designated by the examiner to the examiner.

[0070] Furthermore, below the still image 76 displayed in the balloon window 75, the display control unit 33 displays a numerical value (here, 3 / 4) indicating the number (here, the third) of all still images (here, four) acquired that the still image 76 is, as well as displaying switch buttons 77A and 77B. Here, the switch button 77A is a button for specifying that the still image displayed in the balloon window 75 should be returned to the previous still image, and the switch button 77B is a button for specifying that the still image displayed in the balloon window 75 should be returned to the next still image. By displaying such a user interface, the display control unit 33 can present the still image and classification score at any time designated by the examiner to the examiner.

[0071] (6) Processing Flow Figure 8 is an example of a flowchart executed by the image processing device 1. The image processing device 1 repeatedly executes the processing of this flowchart until the end of the endoscopic examination. Note that, for example, the image processing device 1 determines that the endoscopic examination has ended when it detects a predetermined input to the input unit 14 or the operation unit 36.

[0072] First, the lesion image acquisition unit 30 of the image processing device 1 acquires an endoscopic image Ia (step S11). In this case, the lesion image acquisition unit 30 of the image processing device 1 receives the endoscopic image Ia from the endoscope 3 via the interface 13. In addition, the display control unit 33 executes processing such as displaying the endoscopic image Ia acquired in step S11 on the display device 2.

[0073] Next, the lesion image acquisition unit 30 of the image processing device 1 determines whether a lesion image has been acquired (step S12). In this case, the lesion image acquisition unit 30 acquires, as a lesion image, an endoscopic image Ia designated by the examiner using the operation unit 36 ​​or an endoscopic image Ia in which a lesion site is detected by a lesion detection model. If the lesion image acquisition unit 30 has not acquired a lesion image (step S12; No), the process returns to step S11.

[0074] On the other hand, if the lesion image acquisition unit 30 determines that a lesion image has been acquired (step S12; Yes), the score calculation unit 31 of the image processing device 1 calculates the classification score of each candidate class at the current processing time based on the lesion images obtained in step S12 at the current processing time and past processing times (step S13).

[0075] In calculating the classification score in step S13, the score calculation unit 31 first acquires, as first calculation information, a lesion image or its feature values ​​previously acquired in this flowchart, and calculates a likelihood ratio based on the first calculation information and the lesion image at the current processing time acquired in step S12. The score calculation unit 31 then stores the calculated likelihood ratio and the lesion image or its feature values ​​acquired in step S12 in the first calculation information storage unit D1 as first calculation information. The score calculation unit 31 then calculates an integrated likelihood ratio based on Equation (1) by referring to the likelihood ratios stored in the first calculation information storage unit D1, and determines the calculated integrated likelihood ratio or a function with the integrated likelihood ratio as a variable as the classification score. The score calculation unit 31 also stores the calculated classification score and the like in the second calculation information storage unit D2 as second calculation information. The display control unit 33 may also perform processing such as displaying information related to the classification score calculated by the score calculation unit 31 on the display device 2.

[0076] Next, the classification unit 32 of the image processing device 1 determines whether the classification score of any of the candidate classes reaches the threshold Th (step S14). If the classification unit 32 determines that the classification score of any of the candidate classes reaches the threshold Th (step S14; Yes), it determines a classification of the lesion site. Then, the display control unit 33 causes the display device 2 to execute a display based on the classification result by the classification unit 32 (step S15). On the other hand, if the classification unit 32 determines that the classification score of any of the candidate classes does not reach the threshold Th (step S14; No), it returns the process to step S11.

[0077] (7) Modifications Next, suitable modifications of the above-described embodiment will be described. The following modifications may be applied in combination to the above-described embodiment.

[0078] (Modification 1) The image processing device 1 may simultaneously detect a lesion site in the endoscopic image Ia and classify the lesion site based on the classification score calculated by the score calculation unit 31 .

[0079] In this case, the lesion image acquisition unit 30 supplies the endoscopic image Ia supplied from the endoscope 3 via the interface 13 to the score calculation unit 31 without selecting a lesion image. The score calculation unit 31 then calculates the classification score for each candidate class. In this case, the score calculation unit 31 sets a class representing the absence of a lesion (also referred to as a "lesion non-detection class") as one of the candidate classes, and calculates the classification score for the lesion non-detection class in the same way as the classification scores for the other candidate classes. Then, when the classification score for the lesion non-detection class reaches the threshold value Th, the classification unit 32 generates a classification result indicating that no lesion is present in the endoscopic image Ia used to calculate the classification score.

[0080] As described above, according to this modification, the image processing device 1 can preferably generate classification results including the presence or absence of a lesion site without selecting a lesion image.

[0081] (Modification 2) The image processing device 1 may process a video image composed of endoscopic images Ia generated during an endoscopic examination after the examination.

[0082] For example, at any time after an examination, when an image to be processed is specified based on user input via the input unit 14, the image processing device 1 repeatedly performs the processing of the flowchart shown in Figure 8 on the time-series endoscopic images Ia that constitute the image, until it is determined that the target image has ended.

[0083] 9 is a block diagram of an image processing device 1X according to a second embodiment. The image processing device 1X includes an acquisition unit 30X, a score calculation unit 31X, and a classification unit 32X. The image processing device 1X may be composed of multiple devices.

[0084] The acquisition means 30X acquires endoscopic images of a subject captured by an imaging unit provided in the endoscope. In this case, the acquisition means 30X may immediately acquire endoscopic images generated by the imaging unit, or may acquire endoscopic images generated in advance by the imaging unit and stored in a storage device at a predetermined timing. Furthermore, the endoscopic images acquired by the acquisition means 30X may be endoscopic images in which a lesion site is present (lesion images in the first embodiment). The acquisition means 30X may be, for example, the lesion image acquisition unit 30 in the first embodiment (including modified examples, the same applies below).

[0085] The score calculation means 31X calculates a score relating to the likelihood of each candidate class corresponding to a type of lesion, into which a group of acquired endoscopic images may be classified. Here, the "group of images" is composed of one or more endoscopic images acquired by the acquisition means 30X. The score calculation means 31X may be, for example, the score calculation unit 31 in the first embodiment.

[0086] The classification unit 32X classifies the image group when it is determined that at least one of the scores has reached a threshold value. The classification unit 32X can be, for example, the classification unit 32 in the first embodiment.

[0087] FIG. 10 is an example of a flowchart showing a processing procedure in the second embodiment. First, the acquisition unit 30X acquires endoscopic images of a subject captured by an imaging unit provided in an endoscope (step S21). The score calculation unit 31X calculates a score relating to the likelihood of each candidate class corresponding to a type of lesion into which a group of acquired endoscopic images are to be classified (step S22). Then, if the classification unit 32X determines that at least one of the scores reaches a threshold (step S23; Yes), it classifies the group of images (step S24). On the other hand, if the classification unit 32X determines that none of the scores reaches a threshold (step S23; No), it returns the process to step S21. Note that if the classification unit 32X determines that none of the scores reaches a threshold, instead of returning the process to step S21, it may additionally perform the process described in the first embodiment. For example, when the number of images in the group of images exceeds a predetermined number, the classification means 32X may decide to classify the images into the candidate class with the score closest to the threshold, or may initialize the group of images and restart the processing of the flowchart.

[0088] According to the second embodiment, the image processing device 1X can accurately classify lesion sites present in an endoscopic image.

[0089] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor, etc. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and random access memories (RAMs). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0090] In addition, part or all of the above-described embodiments (including modified examples, the same applies below) can be described as, but are not limited to, the following supplementary notes.

[0091] [Supplementary Note 1] An image processing device comprising: an acquisition means for acquiring endoscopic images of a subject captured by an imaging unit provided in an endoscope; a score calculation means for calculating a score relating to the likelihood of each candidate class corresponding to a type of lesion into which a group of acquired endoscopic images are to be classified; and a classification means for classifying the group of images when it is determined that at least one of the scores has reached a threshold. [Supplementary Note 2] The image processing device according to Supplementary Note 1, wherein, when it is determined that none of the scores has reached the threshold, the score calculation means updates the score based on the group of images to which endoscopic images acquired after the determination have been added, and the classification means classifies the group of images when it is determined that at least one of the updated scores has reached the threshold. [Supplementary Note 3] The image processing device according to Supplementary Note 1 or 2, wherein, when it is determined that none of the scores has reached the threshold and the number of endoscopic images included in the group of images has reached a predetermined number, the classification means outputs the classification result indicating the candidate class corresponding to the score closest to the threshold. [Supplementary Note 4] The image processing device according to Supplementary Note 1 or 2, wherein, when it is determined that none of the scores reaches the threshold value and the number of endoscopic images included in the image group reaches a predetermined number, the score calculation means calculates the score based on the image group of endoscopic images acquired after the number of endoscopic images reaches the predetermined number. [Supplementary Note 5] The image processing device according to Supplementary Note 1, wherein the acquisition means acquires, from endoscopic images output by the imaging unit, lesion images that are endoscopic images containing a lesion site suspected of being the lesion, and the score calculation means calculates the score based on the image group of the lesion images. [Supplementary Note 6] The image processing device according to Supplementary Note 1, further comprising output control means that outputs information related to the score and the classification result via a display device or a sound output device. [Supplementary Note 7] The image processing device according to Supplementary Note 6, wherein the output control means displays a diagram showing the transition of the score for each candidate class on the display device. [Supplementary Note 8] The image processing device according to Supplementary Note 7, wherein the output control means displays, on the display device, the endoscopic images in time series included in the image group in association with the diagram.[Supplementary Note 9] The image processing device according to Supplementary Note 7, wherein the output control means displays the endoscopic image and the score corresponding to the time specified in the figure on the display device. [Supplementary Note 10] An image processing method, in which a computer acquires endoscopic images of a subject captured by an imaging unit provided in an endoscope, calculates a score related to the likelihood of each of candidate classes corresponding to types of lesions into which a group of acquired endoscopic images will be classified, and classifies the group of images if it is determined that at least one of the scores has reached a threshold. [Supplementary Note 11] A storage medium storing a program that causes a computer to execute a process of acquiring endoscopic images of a subject captured by an imaging unit provided in an endoscope, calculates a score related to the likelihood of each of candidate classes corresponding to types of lesions into which a group of acquired endoscopic images will be classified, and classifies the group of images if it is determined that at least one of the scores has reached a threshold.

[0092] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.

[0093] REFERENCE SIGNS LIST 1, 1X image processing device 2 display device 3 endoscope 11 processor 12 memory 13 interface 14 input unit 15 light source unit 16 sound output unit 100 endoscopic examination system

Claims

1. an acquisition means for acquiring an endoscopic image of a subject by an imaging unit provided in the endoscope; a score calculation means for calculating a score relating to the likelihood of each candidate class corresponding to a type of lesion, which is a candidate into which the group of acquired endoscopic images are classified; a classification means for classifying the image group when it is determined that at least one of the scores reaches a threshold value; An image processing device comprising:

2. When it is determined that none of the scores reach the threshold value, the score calculation means updates the scores based on the image group to which the endoscopic image acquired after the determination is added, and The image processing device according to claim 1 , wherein the classification means classifies the group of images when it is determined that at least one of the updated scores has reached the threshold value.

3. 3. The image processing device according to claim 1, wherein when it is determined that none of the scores reach the threshold value and the number of the endoscopic images included in the image group reaches a predetermined number, the classification means outputs the classification result indicating the candidate class corresponding to the score that is closest to the threshold value.

4. The image processing device according to claim 1 or 2, wherein when it is determined that none of the scores reach the threshold value and the number of the endoscopic images included in the image group reaches a predetermined number, the score calculation means calculates the score based on the image group of the endoscopic images acquired after the number of the endoscopic images reaches the predetermined number.

5. the acquiring means acquires, from among the endoscopic images output by the imaging unit, a lesion image which is an endoscopic image including a lesion site suspected of being the lesion; The image processing device according to claim 1 , wherein the score calculation means calculates the score based on a group of the lesion images.

6. The image processing apparatus according to claim 1 , further comprising an output control means for outputting information relating to the scores and the results of the classification by a display device or a sound output device.

7. The image processing apparatus according to claim 6 , wherein the output control means causes the display device to display a diagram showing the transition of the score for each of the candidate classes.

8. The image processing apparatus according to claim 7 , wherein the output control means displays the time-series endoscopic images included in the image group on the display device in association with the diagram.

9. The image processing apparatus according to claim 7 , wherein the output control means displays, on the display device, the endoscopic image and the score corresponding to a time designated in the diagram.

10. The computer An endoscopic image of the subject is obtained by an imaging unit provided in the endoscope; Calculating a score relating to the likelihood of each candidate class corresponding to a type of lesion into which the group of acquired endoscopic images are to be classified; If it is determined that at least one of the scores reaches a threshold, the image group is classified. Image processing methods.

11. An endoscopic image of the subject is obtained by an imaging unit provided in the endoscope; Calculating a score relating to the likelihood of each candidate class corresponding to a type of lesion into which the group of acquired endoscopic images are to be classified; A program that causes a computer to execute a process of classifying the group of images when it is determined that at least one of the scores has reached a threshold value.