Information processing device, information processing method, and recording medium

Abstract

In this information processing device, a lesion detection means detects a lesion that is included in a first endoscope video and a lesion that is included in a second endoscope video. A symptom assessment means acquires a first score that corresponds to a symptom of the lesion detected from the first endoscope video, and a second score that corresponds to a symptom of the lesion detected from the second endoscope video. An identity determination means determines the identity between the lesion detected from the first endoscope video and the lesion detected from the second endoscope video. A change degree assessment means assesses, on the basis of a difference value acquired from the first and the second scores, the change of the symptoms of the identical lesion included in both of the first and the second endoscope videos. The information processing device can be used for decision making by a user who diagnoses a lesion.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to a technique that can be used to present information related to endoscopic examinations.

[0002] Techniques for presenting information related to endoscopic examinations have been proposed in the past.

[0003] Specifically, for example, Patent Document 1 discloses a technology for determining a follow-up date corresponding to the date of follow-up observation of the examination site that was the subject of an endoscopic examination, and presenting the determined follow-up date.

[0004] JP 2016-24542 A

[0005] However, the technique disclosed in Patent Document 1 has a problem in that, for example, it is not possible to grasp the temporal change in the state of a lesion in an examination site that is the target of endoscopic examination.

[0006] One object of the present disclosure is to provide an information processing device capable of presenting information indicating changes over time in the state of a lesion in an examination site that is the target of an endoscopic examination.

[0007] In one aspect of the present disclosure, an information processing device includes: a lesion detection means for detecting at least one lesion included in a first endoscopic image of an examination site of a subject during a first endoscopic examination, and at least one lesion included in a second endoscopic image of the examination site of the subject during a second endoscopic examination performed after the first endoscopic examination; a symptom evaluation means for acquiring a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic image, and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic image; an identity determination means for determining the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image; and a change degree evaluation means for evaluating a change in the symptom of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

[0008] In another aspect of the present disclosure, an information processing method detects at least one lesion included in a first endoscopic image of an examination site of a subject during a first endoscopic examination and at least one lesion included in a second endoscopic image of the examination site of the subject during a second endoscopic examination performed after the first endoscopic examination, obtains a first score indicating an evaluation of symptoms of the lesion detected from the first endoscopic image and a second score indicating an evaluation of symptoms of the lesion detected from the second endoscopic image, determines the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image, and evaluates changes in symptoms of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

[0009] In yet another aspect of the present disclosure, a recording medium records a program that causes a computer to execute a process of detecting at least one lesion included in a first endoscopic image of an examination site of a subject during a first endoscopic examination and at least one lesion included in a second endoscopic image of the examination site of the subject during a second endoscopic examination that is performed after the first endoscopic examination, obtaining a first score indicating an evaluation of symptoms of the lesion detected from the first endoscopic image and a second score indicating an evaluation of symptoms of the lesion detected from the second endoscopic image, determining the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image, and evaluating changes in symptoms of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

[0010] According to the present disclosure, it is possible to present information indicating changes over time in the state of a lesion in an examination site that is the target of an endoscopic examination.

[0011] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to the present disclosure. FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device according to the present disclosure. FIG. 2 is a diagram for explaining an example of a lesion imaging period in an endoscopic video. FIG. 3 is a diagram for explaining another example of a lesion imaging period in an endoscopic video. FIG. 4 is a diagram for explaining an example of a process for extracting an endoscopic image from a lesion imaging section in an endoscopic video. FIG. 5 is a diagram for explaining another example of a process for extracting an endoscopic image from a lesion imaging section in an endoscopic video. FIG. 6 is a diagram for explaining an example of a lesion symptom score. FIG. 7 is a diagram for explaining an example of a method for calculating a difference value of a lesion symptom score. FIG. 8 is a diagram showing a specific example of a difference value of a lesion symptom score. FIG. 9 is a diagram showing a specific example of a lesion symptom score. FIG. 10 is a diagram showing an example of a display image generated by a process according to the present disclosure. FIG. 11 is a diagram showing another example of a display image generated by a process according to the present disclosure. FIG. 12 is a diagram showing another example of a display image generated by a process according to the present disclosure. FIG. 13 is a flowchart showing an example of a process performed in an information processing device according to the present disclosure. FIG. 14 is a diagram showing a modified example of the configuration of a lesion evaluation unit. FIG. 15 is a diagram showing an example of a lesion size score. FIG. 16 is a diagram showing a specific example of a difference value of a lesion size score. Fig. 1 is a diagram showing another example of a display image generated by the processing related to the present disclosure. Fig. 2 is a diagram showing a modified example of the configuration of a lesion evaluation unit. Fig. 3 is a diagram showing an example of an inflammatory score. Fig. 4 is a diagram showing a specific example of a difference value of the inflammatory score. Fig. 4 is a diagram showing another example of a display image generated by the processing related to the present disclosure. Fig. 5 is a block diagram showing another example of the functional configuration of an information processing device related to the present disclosure. Fig. 6 is a flowchart showing another example of processing performed in an information processing device related to the present disclosure.

[0012] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0013] First Embodiment [Hardware Configuration] Fig. 1 is a block diagram showing an example of the hardware configuration of an information processing device according to the present disclosure. As shown in Fig. 1, the information processing device 100 includes an interface (IF) 111, a processor 112, a memory 113, a recording medium 114, a database (DB) 115, a display device 116, and an input device 117. The information processing device 100 can be used, for example, to assist a user in making a decision to diagnose a lesion.

[0014] The IF 111 inputs and outputs data to and from external devices. For example, an endoscopic image obtained by capturing an image of a subject inside a body cavity with an endoscope is input to the information processing device 100 via the IF 111.

[0015] The processor 112 is a computer such as a CPU (Central Processing Unit), and executes a program prepared in advance to control the entire information processing device 100. Specifically, the processor 112 performs, for example, processing to detect a lesion included in an endoscopic image and processing related to the evaluation of the lesion.

[0016] The memory 113 is configured by a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The memory 113 is also used as a working memory while the processor 112 is executing various processes.

[0017] The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or a semiconductor memory, and is configured to be detachable from the information processing device 100. The recording medium 114 records various programs to be executed by the processor 112. When the information processing device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

[0018] The DB 115 stores, for example, endoscopic images input via the IF 111 and processing results obtained by processing by the processor 112.

[0019] The display device 116 is configured by, for example, a liquid crystal monitor, etc. Furthermore, the display device 116 displays a display image including the evaluation result of the lesion, etc., as necessary.

[0020] The input device 117 includes at least one of a keyboard, a mouse, a touch panel, etc. The input device 117 is used to operate a GUI (Graphical User Interface) on a display screen, etc. The input device 117 can output instruction signals corresponding to user operations performed on the GUI, etc.

[0021] [Functional Configuration] Fig. 2 is a block diagram showing an example of the functional configuration of an information processing device according to the present disclosure. As shown in Fig. 2, the information processing device 100 includes a lesion detection unit 11, an image extraction unit 12, a lesion evaluation unit 13, and a display image generation unit 14. In the following description, unless otherwise specified, the examination site to be subjected to the endoscopic examination will be described as being the inside of the large intestine. In the following description, the endoscopic image obtained by capturing an image of a subject inside the large intestine with an endoscope will be described as being input to the information processing device 100.

[0022] The lesion detection unit 11 functions as a lesion detection means. The lesion detection unit 11 also performs processing to detect lesions such as tumors for each endoscopic image EG included in the endoscopic video EV input to the information processing device 100. In the present disclosure, unless otherwise specified, the explanation will be given assuming that information indicating the date and time of the endoscopic examination is attached to the endoscopic video. In the present disclosure, it is also assumed that information similar to the date and time of the endoscopic examination attached to the endoscopic video may be attached to the endoscopic image, etc., as needed.

[0023] Specifically, the lesion detection unit 11 detects lesions included in the endoscopic image EG by inputting the endoscopic image EG included in the endoscopic video EV into the machine learning model MLA. The lesion detection unit 11 can use, as the machine learning model MLA, a model that is trained to output the position of a lesion in an image in response to input of the image including the lesion. It is desirable that the machine learning model MLA be trained using, for example, a plurality of images in which the lesion is positioned differently.

[0024] The lesion detection unit 11 acquires a lesion imaging period DP corresponding to the period during which the lesion detected from the endoscopic image EG was imaged. In other words, each endoscopic image EG corresponding to the lesion imaging period DP in the endoscopic image EV includes a lesion discovered during the endoscopic examination corresponding to the endoscopic image EV. The lesion detection unit 11 also outputs the endoscopic image EV and the lesion imaging period DP to the image extraction unit 12.

[0025] According to the present disclosure, the lesion detection unit 11 can acquire, as the lesion imaging period DP, information indicating the elapsed time from the beginning of the endoscopic video EV, such as information expressed in the format of "X hours, Y minutes, Z seconds." In such a case, the lesion detection unit 11 can acquire, as the lesion imaging period DP, information such as "from 0 hours, 9 minutes, 30 seconds to 0 hours, 10 minutes, 10 seconds."

[0026] The image extraction unit 12 functions as an image extraction means. Based on the lesion photography period DP, the image extraction unit 12 identifies a section in the endoscopic video EV during which lesion photography is continuously performed as a lesion photography section DK. The image extraction unit 12 also extracts at least one endoscopic image EH having relatively high image quality from each endoscopic image EG included in the lesion photography section DK.

[0027] Specifically, the image extraction unit 12 extracts, as the endoscopic image EH, an image that does not contain foreign matter such as water droplets, residue, or treatment tools, is free of blur, and has relatively little blocked-up shadows or blown-out highlights. When extracting the endoscopic image EH, the image extraction unit 12 preferably excludes, for example, endoscopic images EG in which the proportion of low-frequency components in the image is equal to or greater than a predetermined value, as images that have blur. Furthermore, when extracting the endoscopic image EH, the image extraction unit 12 preferably excludes, for example, endoscopic images EG in which the brightness histogram of each pixel in the image is biased around the minimum value, as images with relatively much blocked-up shadows. Furthermore, when extracting the endoscopic image EH, the image extraction unit 12 preferably excludes, for example, endoscopic images EG in which the brightness histogram of each pixel in the image is biased around the maximum value, as images with relatively much blocked-up highlights. Furthermore, when the image extraction unit 12 identifies a plurality of lesion photography sections DK based on the lesion photography period DP, the image extraction unit 12 performs processing to extract an endoscopic image EH for each of the plurality of lesion photography sections DK.

[0028] The image extraction unit 12 acquires a lesion imaging time DT corresponding to the time at which an endoscopic image EH including a lesion was obtained in an endoscopic examination corresponding to the endoscopic video EV, based on the lesion imaging period DP. The image extraction unit 12 also generates information associating the endoscopic image EH and the lesion imaging time DT for each lesion imaging section DK as lesion information DJ corresponding to one endoscopic examination, and outputs the generated lesion information DJ to the lesion evaluation unit 13.

[0029] According to the present disclosure, the image extraction unit 12 can acquire, as the lesion imaging time DT, information in the same format as the lesion imaging period DP. Therefore, the image extraction unit 12 can acquire, for example, information such as "0 hours, 9 minutes, 55 seconds" as the lesion imaging time DT.

[0030] For example, when multiple pieces of lesion information DJ corresponding to multiple endoscopic examinations performed on the same subject on different dates and times are input, the lesion evaluation unit 13 performs processing related to evaluation of the lesions included in each of the endoscopic images EH of the multiple pieces of lesion information DJ. Furthermore, the lesion evaluation unit 13 outputs the multiple pieces of lesion information DJ and evaluation results ER of the lesions included in each of the endoscopic images EH of the multiple pieces of lesion information DJ to the display image generation unit 14. Furthermore, the lesion evaluation unit 13 has a symptom evaluation unit 131, an identity determination unit 132, and a change degree evaluation unit 133.

[0031] The symptom evaluation unit 131 functions as a symptom evaluation unit and acquires a lesion symptom score SC indicating an evaluation of the symptoms of the lesion included in the endoscopic image EH of the lesion information DJ.

[0032] Specifically, the symptom evaluation unit 131 inputs, for example, an endoscopic image EH of the lesion information DJ into the machine learning model MLB to obtain a lesion symptom score SC indicating an evaluation of the symptoms of the lesion included in the endoscopic image EH. The symptom evaluation unit 131 can use, as the machine learning model MLB, a model trained to output a score in response to input of an image including a lesion, where the sum of the probabilities that the lesion corresponds to each of multiple symptoms with different malignancies is 100%. It is desirable that the machine learning model MLB be trained using, for example, multiple images of lesions with different symptoms. Furthermore, when the above-described machine learning model MLB is used, the symptom evaluation unit 131 can obtain, as the lesion symptom score SC, a score including the probabilities of multiple items corresponding to each of multiple symptoms with different malignancies. Furthermore, the symptom evaluation unit 131 obtains a lesion symptom score SC for each of the multiple endoscopic images EH of the lesion information DJ.

[0033] The identity determination unit 132 functions as an identity determination unit, and determines the identity of the lesions included in each of the endoscopic images EH of the multiple pieces of lesion information DJ.

[0034] Specifically, the identity determination unit 132 extracts one endoscopic image EH from each of the multiple lesion information DJ and inputs each extracted endoscopic image EH into the machine learning model MLC to obtain a probability P B indicating the identity of the lesions contained in each of the endoscopic images EH. The identity determination unit 132 can use, as the machine learning model MLC, a model trained to output a probability indicating the identity of the lesions contained in multiple images in response to input of multiple images including lesions that worsen over time. It is desirable that the machine learning model MLC be trained using, for example, images of the inside of the large intestine in which regions corresponding to lesions that worsen over time are masked. Furthermore, the identity determination unit 132 calculates the difference between the lesion imaging times DT contained in each of the multiple lesion information DJ as the time interval T I . In the following description, the time interval calculated by the identity determination unit 132 is assumed to be equal to or greater than zero.

[0035] The identity determination unit 132 obtains a determination result DRS indicating that the lesions included in each endoscopic image EH input to the machine learning model MLC are the same if the probability PB is equal to or greater than a predetermined value THA and the time interval TI is equal to or less than a predetermined time THJ. Furthermore, the identity determination unit 132 determines that the lesions included in each endoscopic image EH are different if the probability PB is less than the predetermined value THA or the time interval TI exceeds the predetermined time THJ.

[0036] If all of the multiple probabilities PB obtained from the combination of an endoscopic image EHX extracted from the lesion information DJX and each endoscopic image EH included in the lesion information DJ corresponding to an endoscopic examination subsequent to the lesion information DJX are less than a predetermined value THA, the identity determination unit 132 obtains a determination result DRC indicating that the lesion included in the endoscopic image EHX is a treated lesion. The lesion information DJX is information generated using the same method as the lesion information DJ and includes an endoscopic image EHX and a lesion imaging time DTX corresponding to at least one lesion imaging section DK. A treated lesion can be described as a lesion that is included in the endoscopic video EV corresponding to an earlier endoscopic examination but not included in the endoscopic video EV corresponding to a later endoscopic examination.

[0037] If all of the multiple probabilities PB obtained from the combination of an endoscopic image EHY extracted from the lesion information DJY and each endoscopic image EH in the lesion information DJ corresponding to an endoscopic examination earlier than the lesion information DJY are less than a predetermined value THA, the identity determination unit 132 obtains a determination result DRN indicating that the lesion included in the endoscopic image EHY is a new lesion. The lesion information DJY is information generated using the same method as the lesion information DJ and includes an endoscopic image EHY and a lesion imaging time DTY corresponding to at least one lesion imaging section DK. A new lesion can be described as a lesion that is not included in the endoscopic video EV corresponding to an earlier endoscopic examination but is included in the endoscopic video EV corresponding to a later endoscopic examination.

[0038] Instead of performing processing using the machine learning model MLC, the identity determination unit 132 may selectively obtain one of the determination results DRS, DRC, and DRN by performing processing such as pattern matching on each endoscopic image EH extracted from multiple lesion information DJ.

[0039] The degree-of-change evaluation unit 133 functions as a degree-of-change evaluation unit, and evaluates the change in symptoms of the lesion LS corresponding to the determination result DRS.

[0040] Specifically, the change degree evaluation unit 133 calculates the difference value DVS by subtracting the lesion symptom score SCX, which indicates an evaluation of the symptoms of a lesion LS included in one endoscopic image EH corresponding to a later endoscopic examination, from the lesion symptom score SCX, which indicates an evaluation of the symptoms of the lesion LS included in another endoscopic image EH corresponding to a later endoscopic examination. In other words, the change degree evaluation unit 133 calculates the difference value DVS, which quantitatively indicates the change in the symptoms of the lesion corresponding to the lesion symptom score SCX and the lesion symptom score SCY. Furthermore, when the lesion symptom scores SCX and SCY include probabilities of multiple items corresponding to multiple symptoms of different malignancies, the change degree evaluation unit 133 calculates the difference value DVS by subtracting each of the multiple items.

[0041] The change degree evaluation unit 133 evaluates the lesion LS based on the difference value DVS to obtain an evaluation result for the lesion LS. Specifically, for the item IMA that shows the largest increase among the items included in the difference value DVS, the change degree evaluation unit 133 obtains the evaluation result for the lesion LS by determining whether the increase in the item IMA is equal to or greater than a threshold value TMA and whether the changes in other items that represent a lower malignancy than the item IMA are negative values. For example, if the increase in the item IMA is equal to or greater than the threshold value TMA and the changes in other items that represent a lower malignancy than the item IMA are negative values, the change degree evaluation unit 133 obtains an evaluation result that the symptoms of the lesion LS corresponding to the difference value DVS tend to progress to the symptoms corresponding to the item IMA. Furthermore, if the increase in the item IMA is less than the threshold value TMA or the changes in other items that represent a lower malignancy than the item IMA are not negative values, the change degree evaluation unit 133 obtains an evaluation result that the symptoms of the lesion LS corresponding to the difference value DVS do not tend to worsen.

[0042] The degree-of-change evaluation unit 133 acquires an evaluation result of the lesion LC corresponding to the determination result DRC for the lesion LC. Specifically, the degree-of-change evaluation unit 133 acquires, for example, an evaluation result indicating that the lesion is a treated lesion as the evaluation result of the lesion LC. That is, the degree-of-change evaluation unit 133 can acquire, as the evaluation result of a lesion that is included in an endoscopic video corresponding to a previous endoscopic examination but is not included in an endoscopic video corresponding to a subsequent endoscopic examination, an evaluation result indicating that the lesion is a treated lesion.

[0043] For a lesion LN corresponding to the determination result DRN, the degree-of-change evaluation unit 133 acquires an evaluation result of the lesion LC corresponding to the determination result DRN. Specifically, the degree-of-change evaluation unit 133 acquires, as the evaluation result of the lesion LN, an evaluation result that the lesion LN is a new lesion having symptoms corresponding to the item IE with the highest probability in the lesion symptom score SC indicating the evaluation of the symptoms of the lesion LN. In other words, the degree-of-change evaluation unit 133 can acquire, as the evaluation result of a lesion that is not included in the endoscopic video corresponding to the earlier endoscopic examination but is included in the endoscopic video corresponding to the later endoscopic examination, an evaluation result that the lesion is a new lesion.

[0044] The change degree evaluation unit 133 obtains an evaluation result ER including at least one of the evaluation result of the lesion LS, the evaluation result of the lesion LC, and the evaluation result of the lesion LN. The change degree evaluation unit 133 also outputs the plurality of pieces of lesion information DJ input to the lesion evaluation unit 13 and the evaluation results ER corresponding to the plurality of pieces of lesion information DJ to the display image generation unit 14.

[0045] The display image generating unit 14 functions as a display image generating means. The display image generating unit 14 generates a display image HG using multiple pieces of lesion information DJ and evaluation results ER, and outputs the generated display image HG to the display device 116.

[0046] [Specific Example] Next, a specific example according to the present disclosure will be described. In this specific example, it is assumed that an endoscopic video EVP corresponding to a previous endoscopic examination performed on a specific subject and an endoscopic video EVC corresponding to a current endoscopic examination performed on the specific subject are input to the information processing device 100.

[0047] The lesion detection unit 11 performs processing to detect lesions such as tumors for each endoscopic image EGP included in the endoscopic video EVP. The lesion detection unit 11 also acquires a lesion imaging period DPP corresponding to the period during which the lesion detected from the endoscopic image EGP was captured. Through this processing, the lesion detection unit 11 can acquire four lesion imaging periods DPP from the endoscopic video EVP, for example, as shown in FIG. 3 . The lesion detection unit 11 also outputs the endoscopic video EVP and the lesion imaging periods DPP to the image extraction unit 12. FIG. 3 is a diagram for explaining an example of a lesion imaging period in an endoscopic video.

[0048] The lesion detection unit 11 performs processing to detect lesions such as tumors for each endoscopic image EGC included in the endoscopic video EVC. The lesion detection unit 11 also acquires a lesion imaging period DPC corresponding to the period during which the detected lesion was captured from the endoscopic image EGC. Through this processing, the lesion detection unit 11 can acquire four lesion imaging periods DPC from the endoscopic video EVC, for example, as shown in FIG. 4 . The lesion detection unit 11 also outputs the endoscopic video EVC and the lesion imaging periods DPC to the image extraction unit 12. FIG. 4 is a diagram for explaining another example of a lesion imaging period in an endoscopic video.

[0049] As described above, the lesion detection unit 11 can detect at least one lesion contained in the endoscopic video EVP obtained by capturing the examination area of ​​a subject during a previous endoscopic examination, and at least one lesion contained in the endoscopic video EVC obtained by capturing the examination area of ​​the same subject during a subsequent endoscopic examination.

[0050] The image extraction unit 12 identifies a section in the endoscopic video EVP during which lesions are continuously photographed as a lesion photography section DKP based on the lesion photography section DPP. The image extraction unit 12 also extracts at least one endoscopic image EHP having relatively high image quality from each endoscopic image EGP included in the lesion photography section DKP. Through this process, the image extraction unit 12 can extract, as the endoscopic image EHP, an image that does not contain foreign matter such as water droplets, residue, or treatment tools, is not blurred, and has relatively little blackout or whiteout. Through the above-described process, the image extraction unit 12 can identify lesion photography sections DKP1 to DKP4 corresponding to the four lesion photography sections DPP, as shown in FIG. 5, for example. Furthermore, according to the above-described processing, the image extraction unit 12 can extract an endoscopic image EHP1 from the lesion shooting section DKP1, an endoscopic image EHP2 from the lesion shooting section DKP2, an endoscopic image EHP3 from the lesion shooting section DKP3, and an endoscopic image EHP4 from the lesion shooting section DKP4 (see Figure 5). Figure 5 is a diagram for explaining an example of processing for extracting endoscopic images from lesion shooting sections in endoscopic video.

[0051] Based on the lesion imaging period DPP, the image extraction unit 12 acquires the lesion imaging time DTP corresponding to the time when the endoscopic image EHP including the lesion was obtained in the endoscopic examination corresponding to the endoscopic video EVP. Through this processing, the image extraction unit 12 can obtain the lesion imaging time DTP1 corresponding to the endoscopic image EHP1, the lesion imaging time DTP2 corresponding to the endoscopic image EHP2, the lesion imaging time DTP3 corresponding to the endoscopic image EHP3, and the lesion imaging time DTP4 corresponding to the endoscopic image EHP4 (see FIG. 5 ).

[0052] The image extraction unit 12 generates information associating the endoscopic image EHP and the lesion photography time DTP for each lesion photography section DKP as lesion information DJP corresponding to the previous endoscopic examination, and outputs the generated lesion information DJP to the lesion evaluation unit 13.

[0053] The image extraction unit 12 identifies a section in the endoscopic video EVC during which lesions are continuously photographed as a lesion photography section DKC based on the lesion photography period DPC. The image extraction unit 12 also extracts at least one endoscopic image EHC having relatively high image quality from each endoscopic image EGC included in the lesion photography section DKC. Through this process, the image extraction unit 12 can extract, as the endoscopic image EHC, an image that does not contain foreign matter such as water droplets, residue, or treatment tools, is not blurred, and has relatively little blackout or whiteout. Through the above-described process, the image extraction unit 12 can identify lesion photography sections DKC1 to DKC4 corresponding to the four lesion photography periods DPC, for example, as shown in FIG. 6 . Furthermore, according to the above-described processing, the image extraction unit 12 can extract an endoscopic image EHC1 from the lesion photography section DKC1, an endoscopic image EHC2 from the lesion photography section DKC2, an endoscopic image EHC3 from the lesion photography section DKC3, and an endoscopic image EHC4 from the lesion photography section DKC4 (see Figure 6). Figure 6 is a diagram for explaining another example of the processing for extracting endoscopic images from the lesion photography sections in the endoscopic video.

[0054] The image extraction unit 12 acquires the lesion imaging time DTC corresponding to the time when the endoscopic image EHC including the lesion was obtained in the endoscopic examination corresponding to the endoscopic video EVC based on the lesion imaging period DPC. Through this processing, the image extraction unit 12 can obtain the lesion imaging time DTC1 corresponding to the endoscopic image EHC1, the lesion imaging time DTC2 corresponding to the endoscopic image EHC2, the lesion imaging time DTC3 corresponding to the endoscopic image EHC3, and the lesion imaging time DTC4 corresponding to the endoscopic image EHC4 (see FIG. 6 ).

[0055] The image extraction unit 12 generates information associating the endoscopic image EHC and the lesion shooting time DTC for each lesion shooting section DKC as lesion information DJC corresponding to the current endoscopic examination, and outputs the generated lesion information DJC to the lesion evaluation unit 13.

[0056] The symptom evaluation unit 131 inputs the endoscopic images EHP of the lesion information DJP into the machine learning model MLB to obtain lesion symptom scores SCP indicating evaluations of the symptoms of the lesions included in the endoscopic images EHP. Through this processing, the symptom evaluation unit 131 can obtain a lesion symptom score SCP1 corresponding to the endoscopic image EHP1, a lesion symptom score SCP2 corresponding to the endoscopic image EHP2, a lesion symptom score SCP3 corresponding to the endoscopic image EHP3, and a lesion symptom score SCP4 corresponding to the endoscopic image EHP4. Furthermore, through the above-described processing, the symptom evaluation unit 131 can obtain, as lesion symptom scores SCP1 to SCP4, scores indicating the probability of four items corresponding to four types of symptoms, as shown in FIG. 7 . FIG. 7 is a diagram showing an example of lesion symptom scores.

[0057] According to the lesion symptom score in FIG. 7 , the probability that a lesion included in an endoscopic image corresponds to "Hyperplastic" is represented as "xx%." According to the lesion symptom score in FIG. 7 , the probability that a lesion included in an endoscopic image corresponds to "Sessile Serrated Lesion" is represented as "yy%." According to the lesion symptom score in FIG. 7 , the probability that a lesion included in an endoscopic image corresponds to "Adenoma" is represented as "zz%." According to the lesion symptom score in FIG. 7 , the probability that a lesion included in an endoscopic image corresponds to "Carcinoma" is represented as "ww%." According to the lesion symptom score in FIG. 7 , the sum of "xx%," "yy%," "zz%," and "ww%" is 100%. Furthermore, the lesion symptom scores in FIG. 7 show that the malignancy of the lesion increases in the order of "Hyperplastic," "Sessile Serrated Lesion," "Adenoma," and "Carcinoma."

[0058] The symptom evaluation unit 131 inputs the endoscopic images EHP of the lesion information DJC into the machine learning model MLB to obtain a lesion symptom score SCC indicating an evaluation of the symptoms of the lesion contained in the endoscopic images EHC. According to this processing, the symptom evaluation unit 131 can obtain a lesion symptom score SCC1 corresponding to the endoscopic image EHC1, a lesion symptom score SCC2 corresponding to the endoscopic image EHC2, a lesion symptom score SCC3 corresponding to the endoscopic image EHC3, and a lesion symptom score SCC4 corresponding to the endoscopic image EHC4. According to the above-described processing, the symptom evaluation unit 131 can obtain, as the lesion symptom scores SCC1 to SCC4, scores indicating the probability of four items corresponding to four types of symptoms, as shown in FIG. 7 .

[0059] As described above, the symptom evaluation unit 131 can acquire a lesion symptom score SCP indicating an evaluation of the symptoms of a lesion detected from the endoscopic image EVP, and a lesion symptom score SCC indicating an evaluation of the symptoms of a lesion detected from the endoscopic image EVC.

[0060] The identity determination unit 132 inputs the endoscopic image EHP extracted from the lesion information DJP and the endoscopic image EHC extracted from the lesion information DJC into the machine learning model MLC to obtain a probability PBC indicating the identity of the lesion contained in the endoscopic image EHP and the lesion contained in the endoscopic image EHC. Specifically, for example, when the lesion information DJP includes endoscopic images EHP1 to EHP4 and the lesion information DJC includes endoscopic images EHC1 to EHC4, the identity determination unit 132 obtains 16 probabilities PBC corresponding to each of the 16 combinations. Furthermore, for example, when the lesion information DJP includes lesion imaging times DTP1 to DTP4 and the lesion information DJC includes lesion imaging times DTC1 to DTC4, the identity determination unit 132 calculates 16 time intervals TIC corresponding to each of the 16 combinations.

[0061] The identity determination unit 132 determines the identity of the lesion contained in the endoscopic image EHP and the lesion contained in the endoscopic image EHC based on the probability PBC and time interval TIC corresponding to the combination of the endoscopic image EHP and the endoscopic image EHC input to the machine learning model MLC. According to this determination, the identity determination unit 132 can obtain, for example, a determination result DRS1 indicating that the same lesion LS1 is contained in the endoscopic image EHP1 and the endoscopic image EHC1. According to the aforementioned determination, the identity determination unit 132 can obtain, for example, a determination result DRS2 indicating that the same lesion LS2 is contained in the endoscopic image EHP2 and the endoscopic image EHC2. According to the aforementioned determination, the identity determination unit 132 can obtain, for example, a determination result DRS3 indicating that the same lesion LS3 is contained in the endoscopic image EHP4 and the endoscopic image EHC3. According to the aforementioned determination, the identity determination unit 132 can obtain, for example, a determination result DRC1 indicating that the treated lesion LC1 is contained in the endoscopic image EHP3. Furthermore, according to the above-described determination, the identity determination unit 132 can obtain, for example, a determination result DRN1 indicating that the endoscopic image EHC4 includes a new lesion LN1.

[0062] As described above, the identity determination unit 132 can determine the identity of the lesions based on the output result obtained by inputting the endoscopic images EHP and EHC to the machine learning model MLC, the lesion imaging time DTP at which the endoscopic image EHP was obtained, and the lesion imaging time DTC at which the endoscopic image EHC was obtained. Also, as described above, the identity determination unit 132 can obtain a determination result that the lesion detected from the endoscopic image EVP and the lesion detected from the endoscopic image EVC are the same lesion when the probability PBC is equal to or greater than a predetermined value THA and the time interval TIC is equal to or less than a predetermined time THJ.

[0063] The change degree evaluation unit 133 calculates a difference value DVS for a lesion LS corresponding to the determination result DRS by subtracting the lesion symptom score SCC, which indicates an evaluation of the symptoms of the lesion LS included in the endoscopic image EHC, from the lesion symptom score SCP, which indicates an evaluation of the symptoms of the lesion LS included in the endoscopic image EHP. Specifically, for example, as shown in FIG. 8 , if the lesion symptom scores SCP and SCC have four items, the change degree evaluation unit 133 calculates the difference value DVS by performing subtraction for each of the four items. According to this process, the change degree evaluation unit 133 can calculate a difference value DVS1 for a lesion LS1 corresponding to the determination result DRS1 by subtracting the lesion symptom score SCC1 from the lesion symptom score SCP1. Furthermore, according to the above-described process, the change degree evaluation unit 133 can calculate a difference value DVS2 for a lesion LS2 corresponding to the determination result DRS2 by subtracting the lesion symptom score SCC2 from the lesion symptom score SCP2. Furthermore, according to the above-described processing, the degree-of-change evaluation unit 133 can calculate a difference value DVS3 for the lesion LS3 corresponding to the determination result DRS3 by subtracting the lesion symptom score SCC3 from the lesion symptom score SCP4. Fig. 8 is a diagram for explaining an example of a method for calculating the difference value of the lesion symptom score.

[0064] The change rate evaluation unit 133 evaluates the lesion LS based on the difference value DVS to obtain an evaluation result for the lesion LS. Specifically, for example, when a difference value DVS such as that shown in FIG. 8 is obtained and the threshold TMA is set to +30%, the change rate evaluation unit 133 obtains an evaluation result indicating that the symptoms of the lesion LS corresponding to the difference value DVS tend to progress to adenoma. On the other hand, when a difference value DVS such as that shown in FIG. 9 is obtained and the threshold TMA is set to +30%, the change rate evaluation unit 133 obtains an evaluation result indicating that the symptoms of the lesion LS corresponding to the difference value DVS do not tend to worsen. Through this processing, the change rate evaluation unit 133 can obtain an evaluation result for lesion LS1 corresponding to difference value DVS1, an evaluation result for lesion LS2 corresponding to difference value DVS2, and an evaluation result for lesion LS3 corresponding to difference value DVS3. FIG. 9 is a diagram showing specific examples of difference values ​​of lesion symptom scores.

[0065] As described above, the change degree evaluation unit 133 can evaluate changes in symptoms of the same lesion contained in both the endoscopic images EVP and EVC based on the difference value DVS obtained from the lesion symptom scores SCP and SCC.

[0066] The degree-of-change evaluation unit 133 acquires the evaluation result that the lesion LC1 corresponding to the determination result DRC1 is a treated lesion.

[0067] For example, when a lesion symptom score SCC4 indicating an evaluation related to the symptoms of lesion LN1 as shown in Fig. 10 is obtained, the change degree evaluation unit 133 acquires an evaluation result that a Sessile Serrated Lesion has been newly discovered as the evaluation result of lesion LN1 corresponding to the judgment result DRN1. Fig. 10 is a diagram showing a specific example of a lesion symptom score.

[0068] The degree-of-change evaluation unit 133 obtains an evaluation result ERZ including the evaluation results of lesions LS1, LS2, LS3, LC1, and LN1. The degree-of-change evaluation unit 133 outputs the lesion information DJP, the lesion information DJC, and the evaluation result ERZ to the display image generation unit 14.

[0069] The display image generation unit 14 uses the lesion information DJP, the lesion information DJC, and the evaluation result ERZ to generate five display images HG1 to HG5 corresponding to each of the five evaluation results included in the evaluation result ERZ. Furthermore, the display image generation unit 14 selectively outputs one of the five display images HG1 to HG5 generated as described above to the display device 116, for example, in response to an instruction signal from the input device 117. The display image generation unit 14 may set a display order based on the lesion imaging times DTP1 to DTP4 included in the lesion information DJP and the lesion imaging times DTC1 to DTC4 included in the lesion information DJC, and output one of the five display images HG1 to HG5 to the display device 116 in accordance with the set display order.

[0070] 11 is a diagram showing an example of a display image generated by the processing according to the present disclosure. The display image generation unit 14 generates, for example, a display image HG1 as shown in FIG. 11 as a display image corresponding to the evaluation result of the lesion LS1 included in the evaluation result ERZ. The display image HG1 includes an endoscopic image EHP1 including the lesion LS1, an endoscopic image EHC1 including the lesion LS1, and a comment CS1 corresponding to the evaluation result regarding the change in symptoms of the lesion LS1. By checking the display image HG1, a user such as a doctor can understand that the symptoms of the lesion LS1 included in the endoscopic images EHP1 and EHC1 are worsening to adenoma, for example.

[0071] 12 is a diagram showing another example of a display image generated by the processing according to the present disclosure. The display image generation unit 14 generates, for example, a display image HG2 as shown in FIG. 12 as a display image corresponding to the evaluation result of lesion LS2 included in the evaluation result ERZ. The display image HG2 includes an endoscopic image EHP2 including lesion LS2, an endoscopic image EHC2 including the lesion LS2, and a comment CS2 corresponding to the evaluation result of lesion LS2. By checking the display image HG2, a user such as a doctor can understand that the symptoms of lesion LS2 included in the endoscopic images EHP2 and EHC2 are worsening to a Sessile Serrated Lesion, for example.

[0072] 13 is a diagram showing another example of a display image generated by the processing according to the present disclosure. The display image generation unit 14 generates, for example, a display image HG3 as shown in FIG. 13 as a display image corresponding to the evaluation result of lesion LS3 included in the evaluation result ERZ. The display image HG3 includes an endoscopic image EHP4 including lesion LS3, an endoscopic image EHC3 including the lesion LS3, and a comment CS3 corresponding to the evaluation result of the lesion LS3. By checking the display image HG3, a user such as a doctor can see that the symptoms of lesion LS3 included in the endoscopic images EHP4 and EHC3 have not worsened.

[0073] FIG. 14 is a diagram illustrating another example of a display image generated by the processing according to the present disclosure. The display image generation unit 14 generates, for example, a display image HG4 as shown in FIG. 14 as a display image corresponding to the evaluation result of lesion LC1 included in the evaluation result ERZ. The display image HG4 includes an endoscopic image EHP3 including lesion LC1 and a comment CC1 corresponding to the evaluation result of lesion LC1. By checking the display image HG4, a user such as a doctor can understand, for example, that a procedure to remove lesion LC1 was performed during the period from the previous endoscopic examination to the current endoscopic examination. Note that according to this embodiment, the display image generation unit 14 may switch between displaying and hiding the comment CC1 in the display image HG4 in response to, for example, an instruction signal from the input device 117.

[0074] 15 is a diagram showing another example of a display image generated by the processing according to the present disclosure. The display image generation unit 14 generates, for example, a display image HG5 as shown in FIG. 15 as a display image corresponding to the evaluation result of the lesion LN1 included in the evaluation result ERZ. The display image HG5 includes an endoscopic image EHC4 including the lesion LN1 and a comment CN1 corresponding to the evaluation result of the lesion LN1. By checking the display image HG5, a user such as a doctor can understand, for example, that a new lesion LN1 has occurred during the period from the previous endoscopic examination to the current endoscopic examination.

[0075] [Processing Flow] Next, a description will be given of the flow of processing performed in the information processing device 100. Fig. 16 is a flowchart showing an example of processing performed in the information processing device according to the present disclosure.

[0076] First, the information processing device 100 acquires a plurality of endoscopic images corresponding to a plurality of endoscopic examinations performed on the same subject on different dates and times (step S11).

[0077] Next, the information processing device 100 identifies, for each of the endoscopic images obtained in step S11, a section in which imaging of a lesion was continuously performed as a lesion imaging section (step S12).

[0078] Next, the information processing device 100 extracts endoscopic images having relatively high image quality from among the endoscopic images included in the lesion photographing section identified in step S12 (step S13).

[0079] Next, the information processing device 100 performs processing related to evaluation of lesions included in the endoscopic images extracted in step S13 (step S14). According to the processing of step S14, the information processing device 100 can obtain, for example, evaluation results related to changes in symptoms of the same lesion included in each of the multiple endoscopic videos obtained in step S11. Furthermore, according to the processing of step S14 described above, the information processing device 100 can obtain, for example, an evaluation result in which, among the lesions included in each of the multiple endoscopic videos obtained in step S11, a lesion that is included in the endoscopic video corresponding to an earlier endoscopic examination but not included in the endoscopic video corresponding to a later endoscopic examination is evaluated as a treated lesion. Furthermore, according to the processing of step S14 described above, the information processing device 100 can obtain, for example, an evaluation result in which, among the lesions included in each of the multiple endoscopic videos obtained in step S11, a lesion that is not included in the endoscopic video corresponding to an earlier endoscopic examination but is included in the endoscopic video corresponding to a later endoscopic examination is evaluated as a new lesion.

[0080] Next, the information processing device 100 generates a display image according to the lesion evaluation result obtained by the evaluation in step S14 (step S15), outputs the generated display screen to the display device 116, and then terminates the series of processes.

[0081] As described above, according to this embodiment, for example, when a follow-up observation of a lesion inside the large intestine is performed, it is possible to present information indicating that the symptoms of the lesion are worsening. Therefore, according to this embodiment, it is possible to present information indicating the temporal change in the condition of the lesion at the examination site that is the target of endoscopic examination.

[0082] As described above, according to this embodiment, it is possible to extract endoscopic images having relatively high image quality from among the endoscopic images included in a section in which lesions were continuously photographed, and to evaluate the lesions included in the extracted endoscopic images. Therefore, according to this embodiment, even if, for example, the photographing conditions for lesions discovered in multiple endoscopic examinations performed on the same subject are different for each examination, it is possible to ensure the quality of the evaluation of the lesions.

[0083] As described above, according to this embodiment, for example, when a follow-up observation of a lesion inside the large intestine is performed, information can be presented indicating that a lesion has been removed during the period from the previous endoscopic examination to the current endoscopic examination.

[0084] As described above, according to this embodiment, for example, when a follow-up observation of a lesion inside the large intestine is performed, information can be presented indicating that a new lesion has occurred during the period from the previous endoscopic examination to the current endoscopic examination.

[0085] [Modifications] Modifications of the present embodiment will be described below. For simplicity, specific descriptions of parts to which the above-described processes can be applied will be omitted as appropriate.

[0086] (Variation 1) Fig. 17 is a diagram showing a variation of the configuration of the lesion evaluation unit. Information processing device 100 may have, for example, a lesion evaluation unit 13A as shown in Fig. 17 instead of lesion evaluation unit 13. Lesion evaluation unit 13A has a symptom evaluation unit 131, an identity determination unit 132, a degree of change evaluation unit 133, and a size evaluation unit 134.

[0087] The size evaluation unit 134 acquires a lesion size score SZ indicating an evaluation of the size of the lesion included in the endoscopic image EH of the lesion information DJ.

[0088] Specifically, the size evaluation unit 134 inputs, for example, the endoscopic image EH of the lesion information DJ into the machine learning model MLD to obtain a lesion size score SZ indicating an evaluation of the size of the lesion contained in the endoscopic image EH. The size evaluation unit 134 can use, as the machine learning model MLD, a model trained to output a score in response to input of an image including a lesion, where the sum of the probabilities of the lesion corresponding to each of multiple sizes is 100%. It is desirable that the machine learning model MLD be trained using, for example, multiple images of lesions of different sizes. Furthermore, when the aforementioned machine learning model MLD is used, the size evaluation unit 134 can obtain, as the lesion score SZ, a score including the probabilities of multiple items corresponding to each of the multiple sizes. Furthermore, the size evaluation unit 134 obtains the lesion score SZ for each of the multiple endoscopic images EH of the lesion information DJ.

[0089] According to the above-described processing, the size evaluation unit 134 can acquire, as the lesion size score, scores indicating the probability of four items corresponding to each of the four types of size, as shown in Fig. 18. Fig. 18 is a diagram showing an example of the lesion size score.

[0090] According to the lesion size score in FIG. 18 , the size of the lesion included in the endoscopic image is expressed in units of 1 mm. According to the lesion size score in FIG. 18 , the probability that the size of the lesion included in the endoscopic image is "5 mm or less" is expressed as "aa %." According to the lesion size score in FIG. 18 , the probability that the size of the lesion included in the endoscopic image is "6 to 9 mm" is expressed as "bb %." According to the lesion size score in FIG. 18 , the probability that the size of the lesion included in the endoscopic image is "10 to 19 mm" is expressed as "cc %." According to the lesion size score in FIG. 18 , the probability that the size of the lesion included in the endoscopic image is "20 mm or more" is expressed as "dd %." According to the lesion size score in FIG. 18 , the sum of "aa %, "bb %, "cc %, " and "dd %" is 100%.

[0091] The degree-of-change evaluation unit 133 calculates a difference value DXS by subtracting a lesion size score SZY corresponding to the size of a lesion LSX contained in another endoscopic image EHXB corresponding to a later endoscopic examination from a lesion size score SZX corresponding to the size of the lesion LSX contained in one endoscopic image EHXA corresponding to a previous endoscopic examination. In other words, the degree-of-change evaluation unit 133 calculates a difference value DXS that quantitatively indicates a change in the size of the lesion LSX corresponding to the lesion size score SZX and the lesion size score SZY. Furthermore, when the lesion symptom scores SZX and SZY have four items as shown in FIG. 18 , the degree-of-change evaluation unit 133 calculates the difference value DXS by subtracting each of the four items.

[0092] According to the above-described process, the degree-of-change evaluation unit 133 can obtain, for example, the difference value DXS as shown in Fig. 19. Fig. 19 is a diagram showing a specific example of the difference value of the lesion size score.

[0093] The change degree evaluation unit 133 evaluates the lesion LSX based on the difference value DXS to obtain an evaluation result for the lesion LSX. Specifically, the change degree evaluation unit 133 obtains the evaluation result for the lesion LSX by determining, for an item IMB that shows the largest increase among the items included in the difference value DXS, whether the amount of change in the item IMB is equal to or greater than a threshold value TMB and whether the amount of change in another item that is smaller in size than the item IMB is a negative value. For example, if the amount of change in the item IMB is equal to or greater than the threshold value TMB and the amount of change in another item that is smaller in size than the item IMB is a negative value, the change degree evaluation unit 133 obtains an evaluation result that the size of the lesion LSX corresponding to the difference value DXS tends to increase to the size corresponding to the item IMB. In addition, the change degree evaluation unit 133 obtains an evaluation result that the size of the lesion LSX corresponding to the difference value DXS does not tend to increase, for example, if the change in the item IMB is less than the threshold value TMB, or if the change in another item representing a size smaller than the item IMB is not a negative value.

[0094] 19 is obtained and the threshold TMB is set to +30%, the change degree evaluation unit 133 obtains an evaluation result indicating that the size of the lesion LSX corresponding to the difference value DXS tends to increase to a size of 10 mm or more and less than 20 mm. Furthermore, according to the processing described above, the change degree evaluation unit 133 can obtain, as the evaluation results of the lesion LSX, an evaluation result relating to a change in the symptoms of the lesion LSX and an evaluation result relating to a change in the size of the lesion LSX.

[0095] 20 is a diagram showing another example of a display image generated by the processing according to the present disclosure. The display image generating unit 14 generates, for example, a display image HGX as shown in FIG. 20 as a display image according to the evaluation result of the lesion LSX.

[0096] The display image HGX includes an endoscopic image EHXA including a lesion LSX, an endoscopic image EHXB including the lesion LSX, a comment CSXA corresponding to an evaluation result regarding a change in the symptoms of the lesion LSX, and a comment CSXB corresponding to an evaluation result regarding a change in the size of the lesion LSX. By checking the display image HGX, a user such as a doctor can understand, for example, that the symptoms of the lesion LSX included in the endoscopic images EHXA and EHXB are worsening to adenoma. Furthermore, by checking the display image HGX, a user such as a doctor can understand, for example, that the size of the lesion LSX included in the endoscopic images EHXA and EHXB is enlarging to a size of 10 mm or more but less than 20 mm.

[0097] According to this modification, the display image generating unit 14 may switch between displaying and hiding the comment CSXB of the display image HGX in response to an instruction signal or the like from the input device 117, for example.

[0098] (Variation 2) Figure 21 is a diagram showing a variation of the configuration of the lesion evaluation unit. Information processing device 100 may have, for example, a lesion evaluation unit 13B as shown in Figure 21 instead of lesion evaluation unit 13. Lesion evaluation unit 13B has a symptom evaluation unit 131, an identity determination unit 132, a degree of change evaluation unit 133, and an inflammatory evaluation unit 135. Furthermore, lesion evaluation unit 13B selectively performs the processing of either symptom evaluation unit 131 or inflammatory evaluation unit 135 in response to, for example, an instruction signal from input device 117.

[0099] The inflammatory evaluation unit 135 can evaluate the symptoms of an inflammatory lesion such as IBD (Inflammatory Bowel Disease), etc. The inflammatory evaluation unit 135 also acquires an inflammatory score SI that indicates an evaluation of the symptoms of the inflammatory lesion included in the endoscopic image EH of the lesion information DJ.

[0100] Specifically, the inflammatory evaluation unit 135 inputs, for example, the endoscopic image EH of the lesion information DJ into the machine learning model MLE to obtain an inflammatory score SI indicating an evaluation of the symptoms of the inflammatory lesion included in the endoscopic image EH. The inflammatory evaluation unit 135 can use, as the machine learning model MLE, a model trained to output a score in response to input of an image including an inflammatory lesion, where the sum of the probabilities that the lesion corresponds to each of multiple symptoms of different malignancies is 100%. It is desirable that the machine learning model MLE be trained using, for example, multiple images including inflammatory lesions and with different lesion symptoms. Furthermore, when the above-described machine learning model MLE is used, the inflammatory evaluation unit 135 can obtain, as the inflammatory score SI, a score including the probabilities of multiple items corresponding to each of multiple symptoms of different malignancies. Furthermore, the inflammatory evaluation unit 135 obtains an inflammatory score SI for each of the multiple endoscopic images EH of the lesion information DJ.

[0101] According to the above-described process, the inflammatory evaluation unit 135 can acquire, as the inflammatory score, a score indicating the probability of three items corresponding to each of three types of symptoms, as shown in Fig. 22. Fig. 22 is a diagram showing an example of the inflammatory score.

[0102] According to the inflammatory score in FIG. 22 , the probability that a lesion included in an endoscopic image corresponds to "non-ulcerative colitis / Crohn's disease" is represented as "ee%." According to the inflammatory score in FIG. 22 , the probability that a lesion included in an endoscopic image corresponds to "ulcerative colitis / Crohn's disease" is represented as "ff%." According to the inflammatory score in FIG. 22 , the probability that a lesion included in an endoscopic image corresponds to "cancer" is represented as "gg%." According to the inflammatory score in FIG. 22 , the sum of "ee%," "ff%," and "gg%" is 100%. According to the inflammatory score in FIG. 22 , the malignancy of the lesion increases in the order of "non-ulcerative colitis / Crohn's disease," "ulcerative colitis / Crohn's disease," and "cancer."

[0103] The change rate evaluation unit 133 calculates a difference value DYS by subtracting an inflammatory score SQY indicating an evaluation of the symptoms of the lesion LSY contained in another endoscopic image EHYB corresponding to a later endoscopic examination from an inflammatory score SQX indicating an evaluation of the symptoms of the lesion LSY contained in one endoscopic image EHYA corresponding to a previous endoscopic examination. In other words, the change rate evaluation unit 133 calculates a difference value DQS that quantitatively indicates a change in the symptoms of the lesion LSY corresponding to the inflammatory score SQX and the inflammatory score SQY. Furthermore, when the inflammatory scores SQX and SQY have three items as shown in FIG. 22 , the change rate evaluation unit 133 calculates the difference value DQS by subtracting each of the three items.

[0104] According to the above-described process, the degree-of-change evaluation unit 133 can acquire, for example, the difference value DQS as shown in Fig. 23. Fig. 23 is a diagram showing a specific example of the difference value of the inflammatory score.

[0105] The degree-of-change evaluation unit 133 evaluates the lesion LSY based on the difference value DQS to obtain an evaluation result for the lesion LSY. Specifically, the degree-of-change evaluation unit 133 obtains the evaluation result for the lesion LSY by determining, for the item IMC that shows the largest increase among the items included in the difference value DQS, whether the amount of change in the item IMC is equal to or greater than a threshold value TMC and whether the amounts of change in other items that represent a lower malignancy than the item IMC are negative values. For example, if the amount of change in the item IMC is equal to or greater than the threshold value TMC and the amounts of change in other items that represent a lower malignancy than the item IMC are negative values, the degree-of-change evaluation unit 133 obtains an evaluation result that the symptoms of the lesion LSY corresponding to the difference value DQS tend to transition to the symptoms corresponding to the item IMC. In addition, the change degree evaluation unit 133 obtains an evaluation result that there is no tendency for the symptoms of the lesion LSY corresponding to the difference value DQS to worsen, for example, if the change in the item IMC is less than the threshold value TMC, or if the change in another item that represents a lower degree of malignancy than the item IMCB is not a negative value.

[0106] 23 is obtained and the threshold value TMC is set to +30%, the change rate evaluation unit 133 obtains an evaluation result indicating that the symptoms of the lesion LSY corresponding to the difference value DQS tend to progress to cancer. Furthermore, the change rate evaluation unit 133 can obtain an evaluation result relating to a change in the symptoms of the lesion LSY as the evaluation result of the lesion LSY.

[0107] 24 is a diagram showing another example of a display image generated by the processing according to the present disclosure. The display image generating unit 14 generates, for example, a display image HGY as shown in FIG. 24 as a display image according to the evaluation result of the lesion LSY.

[0108] The display image HGY includes an endoscopic image EHYA including a lesion LSY, an endoscopic image EHYB including the lesion LSY, and a comment CSY corresponding to an evaluation result regarding a change in symptoms of the lesion LSY. By checking the display image HGY, a user such as a doctor can understand, for example, that the symptoms of the lesion LSY included in the endoscopic images EHYA and EHYB are worsening to cancer.

[0109] Second Embodiment FIG. 25 is a block diagram showing another example of the functional configuration of an information processing device according to the present disclosure.

[0110] The information processing device 500 has the same hardware configuration as the information processing device 100. The information processing device 500 also has a lesion detection unit 511, a symptom evaluation unit 512, an identity determination unit 513, and a change degree evaluation unit 514.

[0111] The lesion detection means 511 can be realized, for example, by using the function of the lesion detection unit 11. The symptom evaluation means 512 can be realized, for example, by using the function of the symptom evaluation unit 131. The identity determination means 513 can be realized, for example, by using the function of the identity determination unit 132. The change degree evaluation means 514 can be realized, for example, by using the function of the change degree evaluation unit 133.

[0112] FIG. 26 is a flowchart showing another example of processing performed in the information processing device according to the present disclosure.

[0113] The lesion detection means 511 detects at least one lesion contained in a first endoscopic image of the subject's examination area captured during a first endoscopic examination, and at least one lesion contained in a second endoscopic image of the subject's examination area captured during a second endoscopic examination performed after the first endoscopic examination (step S51).

[0114] The symptom evaluation means 512 obtains a first score indicating an evaluation of the symptoms of the lesion detected from the first endoscopic image and a second score indicating an evaluation of the symptoms of the lesion detected from the second endoscopic image (step S52).

[0115] The identity determining means 513 determines whether the lesion detected from the first endoscopic video is identical to the lesion detected from the second endoscopic video (step S53).

[0116] The change degree evaluation means 514 evaluates the change in symptoms of the same lesion contained in both the first endoscopic image and the second endoscopic image based on the difference value obtained from the first score and the second score (step S54).

[0117] According to this embodiment, it is possible to present information indicating changes over time in the state of a lesion in an examination site that is the target of an endoscopic examination.

[0118] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0119] (Supplementary Note 1) An information processing device comprising: a lesion detection means for detecting at least one lesion included in a first endoscopic video obtained by capturing an examination site of a subject during a first endoscopic examination, and at least one lesion included in a second endoscopic video obtained by capturing the examination site of the subject during a second endoscopic examination subsequent to the first endoscopic examination; a symptom evaluation means for acquiring a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic video, and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic video; an identity determination means for determining the identity of the lesion detected from the first endoscopic video and the lesion detected from the second endoscopic video; and a change degree evaluation means for evaluating a change in the symptom of the same lesion included in both the first endoscopic video and the second endoscopic video, based on a difference value obtained from the first score and the second score.

[0120] (Appendix 2) The information processing device of Appendix 1 further comprises an image extraction means for extracting a first endoscopic image having relatively high image quality from each endoscopic image included in a section in which the lesion was continuously photographed in the first endoscopic video, and extracting a second endoscopic image having relatively high image quality from each endoscopic image included in a section in which the lesion was continuously photographed in the second endoscopic video, wherein the symptom evaluation means obtains the first score corresponding to the first endoscopic image and the second score corresponding to the second endoscopic image.

[0121] (Appendix 3) The information processing device of Appendix 2, wherein the image extraction means extracts, as the first endoscopic image and the second endoscopic image, images that do not contain foreign matter, are not blurred, and have relatively little black crush and white blowout.

[0122] (Appendix 4) The information processing device of the appendix, wherein when extracting the first endoscopic image and the second endoscopic image, the image extraction means excludes endoscopic images in which the proportion of low-frequency components in the image is equal to or greater than a predetermined value as images in which blurring occurs.

[0123] (Appendix 5) The information processing device of Appendix 3, wherein the image extraction means, when extracting the first endoscopic image and the second endoscopic image, excludes endoscopic images that have a bias around the minimum value in the brightness histogram of each pixel in the image as images with relatively more crushed black.

[0124] (Supplementary Note 6) The information processing device of Supplementary Note 3, wherein the image extraction means, when extracting the first endoscopic image and the second endoscopic image, excludes endoscopic images that have a bias around the maximum value in the brightness histogram of each pixel in the image as images with a relatively large amount of blown-out highlights.

[0125] (Appendix 7) The information processing device of Appendix 2, wherein the identity determination means determines the identity based on an output result obtained by inputting the first endoscopic image and the second endoscopic image into a machine learning model, a first time when the first endoscopic image was obtained in the first endoscopic examination, and a second time when the second endoscopic image was obtained in the second endoscopic examination.

[0126] (Appendix 8) The information processing device of Appendix 7, wherein the identity determination means obtains a determination result that the lesion detected from the first endoscopic video and the lesion detected from the second endoscopic video are the same lesion when the probability indicating the identity obtained as the output result is equal to or greater than a predetermined value and the time interval obtained as the difference between the first time and the second time is equal to or less than a predetermined time.

[0127] (Supplementary Note 9) The information processing device of Supplementary Note 7, wherein the machine learning model is trained using an image in which an area corresponding to a lesion that worsens over time in an image of the same region as the examination region is masked.

[0128] (Supplementary Note 10) The information processing device of Supplementary Note 1, wherein the change degree evaluation means obtains an evaluation result indicating that a lesion that is included in the first endoscopic image but not included in the second endoscopic image has been treated.

[0129] (Supplementary Note 11) The information processing device of Supplementary Note 1, wherein the change degree evaluation means obtains an evaluation result indicating that a lesion that is not included in the first endoscopic image but is included in the second endoscopic image is a new lesion.

[0130] (Supplementary Note 12) The information processing device of Supplementary Note 1, wherein the symptom evaluation means obtains, as the first score and the second score, scores including the probabilities of a plurality of items corresponding to each of a plurality of symptoms of different malignancies, and the change degree evaluation means obtains the difference value by performing subtraction for each of the plurality of items.

[0131] (Supplementary Note 13) The information processing device of Supplementary Note 12, wherein the change degree evaluation means obtains an evaluation result indicating that the symptoms of the same lesion tend to progress to the symptoms corresponding to one item that shows the largest increase among the items included in the difference value, when the increase in the one item is equal to or greater than a threshold and the change in another item that shows a lower degree of malignancy than the one item is a negative value.

[0132] (Supplementary Note 14) The information processing device of Supplementary Note 12, wherein the change degree evaluation means obtains an evaluation result that, for one item that shows the largest increase among the items included in the difference value, if the increase in the one item is less than a threshold value, or if the change in another item that shows a lower malignancy than the one item is not a negative value, indicates that there is no tendency for the symptoms of the same lesion to worsen.

[0133] (Supplementary Note 15) The information processing device of Supplementary Note 2 further includes a display image generating means for generating a display image including the first endoscopic image, the second endoscopic image, and a comment corresponding to an evaluation result related to a change in symptoms of the same lesion.

[0134] (Appendix 16) An information processing device of Appendix 1, further comprising a size evaluation means for acquiring a third score indicating an evaluation of the size of the lesion detected from the first endoscopic video and a fourth score indicating an evaluation of the size of the lesion detected from the second endoscopic video, wherein the change degree evaluation means evaluates the change in size of the same lesion included in both the first endoscopic video and the second endoscopic video based on a difference value obtained from the third score and the fourth score.

[0135] (Appendix 17) An information processing device of Appendix 1, further comprising an inflammatory evaluation means for acquiring a fifth score indicating an evaluation of the symptoms of an inflammatory lesion detected from the first endoscopic video and a sixth score indicating an evaluation of the symptoms of an inflammatory lesion detected from the second endoscopic video, wherein the change degree evaluation means evaluates the change in the symptoms of the same lesion included in both the first endoscopic video and the second endoscopic video based on a difference value obtained from the fifth score and the sixth score.

[0136] (Supplementary Note 18) The information processing device of Supplementary Note 1, wherein the examination site in the first endoscopic examination and the second endoscopic examination is the inside of the large intestine.

[0137] (Supplementary Note 19) An information processing method that detects at least one lesion included in a first endoscopic image of an examination site of a subject taken during a first endoscopic examination, and at least one lesion included in a second endoscopic image of the examination site of the subject taken during a second endoscopic examination subsequent to the first endoscopic examination; obtains a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic image, and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic image; determines the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image; and evaluates a change in the symptom of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

[0138] (Supplementary Note 20) A recording medium having recorded thereon a program that causes a computer to execute the following processes: detect at least one lesion included in a first endoscopic image of an examination site of a subject taken during a first endoscopic examination, and at least one lesion included in a second endoscopic image of the examination site of the subject taken during a second endoscopic examination subsequent to the first endoscopic examination; obtain a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic image, and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic image; determine the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image; and evaluate a change in a symptom of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

[0139] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0140] REFERENCE SIGNS LIST 11 Lesion detection unit 12 Image extraction unit 13 Lesion evaluation unit 14 Display image generation unit 100 Information processing device 131 Symptom evaluation unit 132 Identity determination unit 133 Change degree evaluation unit

Claims

1. An information processing device having: a lesion detection means for detecting at least one lesion included in a first endoscopic image captured of an examination site of a subject during a first endoscopic examination, and at least one lesion included in a second endoscopic image captured of the examination site of the subject during a second endoscopic examination subsequent to the first endoscopic examination; a symptom evaluation means for acquiring a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic image and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic image; an identity determination means for determining the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image; and a change evaluation means for evaluating a change in symptoms of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

2. An information processing device as described in claim 1, further comprising an image extraction means for extracting a first endoscopic image having relatively high image quality from among the endoscopic images included in a section in which the lesion was continuously photographed in the first endoscopic video, and extracting a second endoscopic image having relatively high image quality from among the endoscopic images included in a section in which the lesion was continuously photographed in the second endoscopic video, wherein the symptom evaluation means obtains the first score corresponding to the first endoscopic image and the second score corresponding to the second endoscopic image.

3. An information processing device as described in claim 2, wherein the image extraction means extracts, as the first endoscopic image and the second endoscopic image, images that do not contain foreign objects, are not blurred, and have relatively little crushed shadows and blown out highlights.

4. An information processing device as described in claim 3, wherein the image extraction means, when extracting the first endoscopic image and the second endoscopic image, excludes endoscopic images in which a proportion of low-frequency components in the image is equal to or greater than a predetermined value as images in which blurring occurs.

5. An information processing device as described in claim 3, wherein the image extraction means, when extracting the first endoscopic image and the second endoscopic image, excludes endoscopic images that have a bias around the minimum value in the brightness histogram of each pixel in the image as images with relatively high amounts of crushed black.

6. An information processing device as described in claim 3, wherein the image extraction means, when extracting the first endoscopic image and the second endoscopic image, excludes endoscopic images that have a bias around the maximum value in the brightness histogram of each pixel in the image as images with relatively high amounts of blown-out highlights.

7. The information processing device described in claim 2, wherein the identity determination means determines the identity based on an output result obtained by inputting the first endoscopic image and the second endoscopic image into a machine learning model, a first time when the first endoscopic image was obtained in the first endoscopic examination, and a second time when the second endoscopic image was obtained in the second endoscopic examination.

8. An information processing device as described in claim 7, wherein the identity determination means obtains a determination result that the lesion detected from the first endoscopic video and the lesion detected from the second endoscopic video are the same lesion when the probability indicating the identity obtained as the output result is equal to or greater than a predetermined value and the time interval obtained as the difference between the first time and the second time is equal to or less than a predetermined time.

9. The information processing device according to claim 7, wherein the machine learning model is trained using an image in which an area corresponding to a lesion that worsens over time in an image of the same area as the examination area is masked.

10. An information processing device as described in claim 1, wherein the change degree evaluation means obtains an evaluation result that a lesion that is included in the first endoscopic image but not included in the second endoscopic image is a treated lesion.

11. An information processing device as described in claim 1, wherein the change degree evaluation means obtains an evaluation result that a lesion that is not included in the first endoscopic image but is included in the second endoscopic image is a new lesion.

12. The information processing device according to claim 1, wherein the symptom evaluation means obtains, as the first score and the second score, scores including the probabilities of a plurality of items corresponding to each of a plurality of symptoms of differing malignancy, and the change degree evaluation means obtains the difference value by performing subtraction for each of the plurality of items.

13. The information processing device according to claim 12, wherein the change evaluation means obtains an evaluation result indicating that, for a single item showing the largest increase among the items included in the difference value, when the increase in the single item is equal to or greater than a threshold value and the change in other items showing a lower degree of malignancy than the single item is a negative value, the symptoms of the same lesion tend to progress to the symptoms corresponding to the single item.

14. The information processing device according to claim 12, wherein the change evaluation means obtains an evaluation result indicating that there is no tendency for symptoms of the same lesion to worsen when, for an item that shows the largest increase among the items included in the difference value, the increase in the item is less than a threshold value, or when the change in another item that represents a lower degree of malignancy than the item in question is not a negative value.

15. An information processing device as described in claim 2, further comprising a display image generating means for generating a display image including the first endoscopic image, the second endoscopic image, and a comment corresponding to an evaluation result relating to changes in symptoms of the same lesion.

16. An information processing device as described in claim 1, further comprising a size evaluation means for acquiring a third score indicating an evaluation of the size of a lesion detected from the first endoscopic video and a fourth score indicating an evaluation of the size of a lesion detected from the second endoscopic video, wherein the change degree evaluation means evaluates the change in size of the same lesion contained in both the first endoscopic video and the second endoscopic video based on a difference value obtained from the third score and the fourth score.

17. An information processing device as described in claim 1, further comprising an inflammatory evaluation means for acquiring a fifth score indicating an evaluation of a symptom of an inflammatory lesion detected from the first endoscopic image and a sixth score indicating an evaluation of a symptom of an inflammatory lesion detected from the second endoscopic image, wherein the change degree evaluation means evaluates a change in symptoms of the same lesion contained in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the fifth score and the sixth score.

18. An information processing device according to claim 1, wherein the examination site in the first endoscopic examination and the second endoscopic examination is the inside of the large intestine.

19. An information processing method comprising: detecting at least one lesion included in a first endoscopic image of an examination site of a subject during a first endoscopic examination, and at least one lesion included in a second endoscopic image of the examination site of the subject during a second endoscopic examination performed after the first endoscopic examination; obtaining a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic image and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic image; determining the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image; and evaluating a change in symptoms of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

20. A recording medium having a program recorded thereon that causes a computer to execute a process of detecting at least one lesion included in a first endoscopic image of an examination site of a subject during a first endoscopic examination, and at least one lesion included in a second endoscopic image of the examination site of the subject during a second endoscopic examination subsequent to the first endoscopic examination, obtaining a first score indicating an evaluation of a symptom of the lesion detected from the first endoscopic image and a second score indicating an evaluation of a symptom of the lesion detected from the second endoscopic image, determining the identity of the lesion detected from the first endoscopic image and the lesion detected from the second endoscopic image, and evaluating a change in symptoms of the same lesion included in both the first endoscopic image and the second endoscopic image based on a difference value obtained from the first score and the second score.

Images