Explanation support device, explanation support method, program for explanation support, and non-transitory recording medium for recording program for explanation support

Abstract

This explanation support device includes a specification unit and an explanation basis acquisition unit. The specification unit receives, with respect to a first image, a first identification result and a first certainty factor output by a black-box type first inference engine, and, with respect to the first image, a second identification result and a second certainty factor output by a white-box type second inference engine. Furthermore, the specification unit receives at least one identification result and certainty factor from among the following: a third identification result and a third certainty factor output by a white-box type third inference engine different from the second inference engine, with respect to the first image; a fourth identification result and a fourth certainty factor output by the second inference engine, with respect to a second image separated by a predetermined number of frames from the first image; and a fifth identification result and a fifth certainty factor output by the second inference engine, with respect to a third image obtained by processing the first image. The specification unit identifies, from the certainty factors other than the received first certainty factor, a certainty factor closest to the first certainty factor, among the certainty factors associated with the same identification result as the first identification result. The explanation basis acquisition unit acquires identification basis information associated with the identified certainty factor and outputs the identification basis information to a monitor.

Description

Explanation support device, explanation support method, explanation support program, and non-transitory recording medium for recording the explanation support program

[0001] The present invention relates to an explanation support device, an explanation support method, an explanation support program, and a non-transitory recording medium for recording an explanation support program, for providing evidence for inferences made by AI.

[0002] In recent years, technologies that utilize AI (artificial intelligence) based on image data to assist in judgments that were previously made visually by humans have been developed in various fields. For example, in the medical field, Computer Aided Diagnosis (CAD) has been developed, which uses AI to provide identification results such as the differentiation of lesions as support information.

[0003] There is also known a technique for presenting evidence for inferences made by AI. For example, by using a white-box CAD system, it is possible to present evidence for inferences.

[0004] However, CAD AI used in medical settings such as endoscopic diagnosis is configured as a black box, making it difficult to present evidence. While it is possible to configure CAD AI as a white box, white box AI may have lower inference accuracy than black box AI.

[0005] Japanese Patent Application Publication No. 2021-100555 (hereinafter referred to as Patent Document 1) discloses a technology that uses separate AIs for the same endoscopic image, outputs identification results of site information from one AI and lesion information from the other AI, and checks the consistency of this information to confirm whether the AI ​​has made an erroneous judgment. By adopting the technology of Patent Document 1, it becomes possible to determine whether the identification result of the lesion is likely to be accurate.

[0006] Japanese Patent Application Laid-Open No. 2021-100555

[0007] However, even if the technology of Patent Document 1 is used, it is not possible to obtain basis information for the user to determine why the identification result was output. The present invention aims to provide an explanation support device, an explanation support method, an explanation support program, and a non-transitory recording medium for recording the explanation support program, which are capable of obtaining highly accurate identification results using black box-type AI while also obtaining basis information for inference.

[0008] An explanation support device according to one aspect of the present invention receives a first classification result and a first certainty factor output by a first black-box type inference device for a first image, and a second classification result and a second certainty factor output by a second white-box type inference device for the first image, and further receives at least one of a third classification result and a third certainty factor output by a third white-box type inference device different from the second inference device for the first image, a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image, and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image, and includes an identification unit that identifies, from the certainty factors other than the first certainty factor received, a certainty factor closest to the first certainty factor that is associated with the same classification result as the first classification result, and an explanation basis acquisition unit that acquires identification basis information associated with the identified certainty factor and outputs it to a monitor.

[0009] An explanation support device according to one aspect of the present invention includes a processor, which receives a first classification result and a first certainty factor output by a first black-box type inference device for a first image, and a second classification result and a second certainty factor output by a second white-box type inference device for the first image, and further receives at least one of a third classification result and a third certainty factor output by a third white-box type inference device different from the second inference device for the first image, a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image, and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image, and identifies, from the certainty factors other than the received first certainty factor, a certainty factor closest to the first certainty factor that is associated with the same classification result as the first classification result, and obtains identification basis information associated with the identified certainty factor and outputs it to a monitor.

[0010] An explanation assistance method according to one aspect of the present invention receives a first classification result and a first certainty factor output by a first black-box inference device for a first image, and a second classification result and a second certainty factor output by a second white-box inference device for the first image; further receives at least one classification result and certainty factor from among a third classification result and a third certainty factor output by a third white-box inference device different from the second inference device for the first image, a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image, and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image; identifies, from the certainty factors other than the received first certainty factor, a certainty factor closest to the first certainty factor that is associated with the same classification result as the first classification result; and obtains identification basis information associated with the identified certainty factor and outputs it to a monitor.

[0011] An explanation support program according to one aspect of the present invention causes a computer to execute the following steps: receive a first classification result and a first certainty factor output by a first black-box inference device for a first image, and a second classification result and a second certainty factor output by a second white-box inference device for the first image; further receive at least one classification result and certainty factor from a third white-box inference device different from the second inference device for the first image; a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image; and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image; identify, from the certainty factors other than the received first certainty factor, a certainty factor closest to the first certainty factor that is associated with the same classification result as the first classification result; and obtain identification basis information associated with the identified certainty factor and output it to a monitor.

[0012] A non-transitory recording medium for recording an explanation support program according to one aspect of the present invention records on a computer the explanation support program, which executes the following steps: receiving a first classification result and a first certainty factor output by a first black-box type inference device for a first image, and a second classification result and a second certainty factor output by a second white-box type inference device for the first image; further receiving at least one of a third classification result and a third certainty factor output by a third white-box type inference device different from the second inference device for the first image; a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image; and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image; identifying, from the certainty factors other than the received first certainty factor, a certainty factor closest to the first certainty factor that is associated with the same classification result as the first classification result; obtaining identification basis information associated with the identified certainty factor and outputting it to a monitor.

[0013] According to the present invention, it is possible to obtain highly accurate identification results using a black box type AI while also obtaining basis information for inference.

[0014] FIG. 1 is a block diagram showing an explanation support device according to a first embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example in which a white-box reasoner IW is configured using a decision tree. FIG. 3 is a flowchart for explaining the operation of the first embodiment. FIG. 4 is an explanatory diagram for explaining the operation of the first embodiment. FIG. 5 is a block diagram showing a modified example. FIG. 6 is a block diagram showing a second embodiment. FIG. 7 is a block diagram showing a third embodiment. FIG. 8 is a block diagram showing a fourth embodiment. FIG. 9 is an explanatory diagram for explaining the embodiments of FIGS. 6 and 8. FIG. 10 is a block diagram showing a fifth embodiment. FIG. 11 is a block diagram showing a sixth embodiment. FIG. 12 is a block diagram showing a seventh embodiment.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] (First Embodiment) Figure 1 is a block diagram showing an explanation support device according to a first embodiment of the present invention. In this embodiment, a black-box inference device and multiple white-box inference devices are used, and the identification result of the black-box inference device is presented. The identification result and certainty factor of the black-box inference device are compared with the identification results and certainty factors of the multiple white-box inference devices, identification basis information is obtained from a white-box inference device selected based on the comparison result, and the obtained identification basis information is presented. This makes it possible to obtain identification basis information for inference while also obtaining highly accurate identification results.

[0017] FIG. 1 illustrates an example in which this embodiment is applied to AI diagnosis using endoscopic images, but this embodiment is not limited to this and can be applied to various processing circuits using AI.

[0018] The explanation support device 1 includes a control unit 10 and a comparator 12. Furthermore, as shown in FIG. 1 , the explanation support device 1 may include at least one of an image processing circuit 11, a synthesis circuit 13, a black-box-type first inference unit IB, a white-box-type second inference unit IW1, and white-box-type third inference units IW2 and IW3, or these may be located in separate devices. An endoscopic image of the inside of a body captured by an endoscope (not shown) is supplied to the image processing circuit 11 as a first image. The control unit 10 may be configured with a processor using a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or the like. The control unit 10 may operate according to a program stored in a memory (not shown) to control each unit, or may implement some or all of its functions using hardware electronic circuits. The control unit 10 comprehensively controls the entire explanation support device 1.

[0019] The image processing circuit 11 performs predetermined signal processing on the input endoscopic image, such as color adjustment processing, matrix conversion processing, noise removal processing, adaptive processing, and various other signal processing. The image processing circuit 11 outputs the image information (endoscopic image) after signal processing to the synthesis circuit 13, and also outputs it to the first inference unit IB, the second inference unit IW1, and the third inference units IW2 and IW3.

[0020] In FIG. 1 , for convenience, the white-box inference device is divided into a second inference device IW1 and two third inference devices IW2 and IW3. However, in this embodiment, it is sufficient to have one black-box inference device IB and multiple white-box inference devices. This does not merely indicate that the second inference device IW1 and the third inference devices IW2 and IW3 are different types. Furthermore, the number of third inference devices can be set to one or more as appropriate. The first inference device IB is a black-box inference device. The first inference device IB can be configured, for example, by a neural network. For example, the first inference device IB may be constructed by deep learning using a multilayer neural network. The neural network is configured by multiple product-sum operations, and the inference device is constructed by performing learning to adjust the multipliers (weights) of the product-sum operations so that the output, when training data is input to the neural network, approaches the corresponding correct answer information. The inference machine, which is a trained neural network, becomes capable of "identification (inference)" to appropriately derive a solution for unknown input.

[0021] The first inference unit IB performs inference on the input endoscopic image and obtains an identification result and information on a certainty factor as an inference result. For example, the first inference unit IB uses inference to identify the position (area) of a lesion or the like contained in the endoscopic image, obtains information for generating a frame image showing the area, and differentiates the lesion to obtain information on the differentiation result. The first inference unit IB outputs the frame image and differentiation information, which are information on the position (area) of the lesion, as an identification result (hereinafter referred to as a first identification result), and also outputs a certainty factor (hereinafter referred to as a first certainty factor) indicating the certainty (reliability) of the identification result. The first identification result and first certainty factor from the first inference unit IB are supplied to the synthesis circuit 13 as lesion information.

[0022] The second inference unit IW1 and the third inference units IW2 and IW3 (hereinafter, when there is no need to distinguish between these inference units, these inference units will be referred to as white-box inference units IW) are white-box inference units. The white-box inference unit IW has, for example, a configuration in which the relationship between input and output is clearly defined and its internal structure is interpretable. For example, the white-box inference unit IW may employ a model using an algorithm such as a Bayesian network, a decision tree, a linear regression, a pattern matching, a support vector machine, a naive Bayes, a k-NN, or a k-means algorithm. The second inference unit IW1 performs inference on an input endoscopic image and outputs, for example, a classification result indicating the result of distinguishing a lesion (hereinafter, referred to as a second classification result) and its confidence level (hereinafter, referred to as a second confidence level). The third inference units IW2 and IW3 perform inference on an input endoscopic image and output, for example, a classification result indicating the result of distinguishing a lesion (hereinafter, referred to as a third classification result) and its confidence level (hereinafter, referred to as a third confidence level). Furthermore, the white-box reasoner IW outputs basis information (identification basis information) indicating the basis for obtaining each identification result.

[0023] FIG. 2 is an explanatory diagram showing an example of a white-box reasoner IW configured using a decision tree. The example in FIG. 2 shows an example in which color, vascular pattern, and surface pattern of lesions in an endoscopic image are classified at a predetermined node of the decision tree. The example in the left column of FIG. 2 shows an example in which color, vascular pattern, and surface pattern are each classified into three types. The example in the right column of FIG. 2 shows an example in which vascular pattern and surface pattern are each classified into three types, and color is classified into six types. That is, the white-box reasoner IW configured using the decision tree in the left column of FIG. 2 and the white-box reasoner IW configured using the decision tree in the right column of FIG. 2 are trained using training data sets with different samples and feature amounts, resulting in different node divisions. That is, the white-box reasoner IW configured using the decision tree in the left column of FIG. 2 and the white-box reasoner IW configured using the decision tree in the right column of FIG. 2 output classification results according to their respective different characteristics. It can be said that the decision tree in the right column of FIG. 2 has higher accuracy in classification using color than the decision tree in the left column of FIG.

[0024] When the white-box reasoner IW is configured as a decision tree, the classification of each node indicates the basis for identification. For example, the basis for determining that a lesion has a certain color, a certain vascular pattern, and a certain surface pattern is a lesion of a certain disease becomes clear.

[0025] 2, each white-box reasoner IW may have the same algorithm but different machine learning data (training data). When each white-box reasoner IW is configured using a decision tree, the node divisions will be different from each other, and each white-box reasoner IW will have different discrimination characteristics. In other words, each white-box reasoner IW may have different discrimination characteristics from each other and output different discrimination results from each other.

[0026] In this embodiment, it is sufficient to employ a plurality of white-box inference devices IW having mutually different discrimination characteristics as the white-box inference device IW, and it is also possible to employ inference devices configured with different algorithms as the white-box inference device IW.

[0027] In general, it can be said that the classification accuracy (inference accuracy) of the first reasoner IB is sufficiently higher than that of the white-box reasoner IW. The first classification result and first certainty factor from the first reasoner IB are provided to the comparator 12. In addition, each classification result and each certainty factor from each white-box reasoner IW are also provided to the comparator 12.

[0028] The comparator 12 includes an identification unit 12a and an explanation basis acquisition unit 12b. The comparator 12 may be configured by a processor using a CPU, an FPGA, or the like. The comparator 12 may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of its functions using a hardware electronic circuit.

[0029] The identification unit 12a of the comparator 12 compares the first identification result from the first inference unit IB with each identification result from each white-box inference unit IW. The certainty output from a given white-box inference unit IW whose identification result matches the first identification result is referred to as the certainty associated with the same identification result as the first identification result. The identification unit 12a identifies the white-box inference unit IW that outputs an identification result that matches the first identification result from the first inference unit IB, thereby identifying the certainty associated with the same identification result as the first identification result.

[0030] In this embodiment, by employing a plurality of white-box reasoners with different identification characteristics as the white-box reasoners IW, it is possible to obtain a plurality of different identification results. Using a plurality of white-box reasoners IW increases the likelihood that one of the identification results from the white-box reasoners IW will match the first identification result of the first reasoner IB, which has a relatively high identification accuracy (inference accuracy), compared to using a single white-box reasoner IW. When the identification result output by one of the white-box reasoners IW matches the first identification result, it can be determined that the identification result of that white-box reasoner IW is likely to be correct.

[0031] Furthermore, when there are multiple white-box reasoners IW whose identification results match the first identification result of the first reasoner IB, it can be determined that the identification result of a second reasoner IW among the multiple white-box reasoners IW whose output certainty factor is closest to the first certainty factor is more likely to be correct than the identification results of the other second reasoners IW. The identification unit 12a of the comparator 12 identifies the certainty factor closest to the first certainty factor among the certainty factors associated with the same identification result as the first identification result.

[0032] The identification basis information obtained from the white-box reasoner IW that outputs the certainty factor determined by the determination unit 12a is referred to as the identification basis information associated with the determined certainty factor. The explanation basis acquisition unit 12b acquires the identification basis information associated with the determined certainty factor based on the information on the certainty factor determined by the determination unit 12a.

[0033] The comparator 12 outputs the identification basis information acquired by the explanation basis acquisition unit 12b to the synthesis circuit 13. The synthesis circuit 13 synthesizes the image information from the image processing circuit 11, the lesion information from the first inference unit IB, and the basis information from the comparator 12. For example, the synthesis circuit 13 generates display data for a composite display in which images based on the image information, lesion information, and basis information are synthesized. The synthesis circuit 13 provides the generated display data to a monitor MO for display. The monitor MO is composed of a display device such as an LCD (liquid crystal display device), receives the display data from the synthesis circuit 13, and displays the composite display on its display screen.

[0034] In the example of Fig. 1, the synthesis circuit 13 displays an endoscopic image P1 based on image information on the left side of the display screen of the monitor MO. The synthesis circuit 13 also displays a frame image P2 indicating the area of ​​the lesion (hatched area) in the endoscopic image P1 based on the lesion information, and displays "cancer I 80%" below the endoscopic image P1, indicating that the classification result is stage I cancer with a certainty factor of 80%. The synthesis circuit 13 also displays a basis image P4 based on the basis information on the right side of the display screen of the monitor MO. The basis image P4 indicates the basis of the classification by using colors, etc. (hatched areas) to indicate which branches were selected for each classification in the decision tree, and by using colors, etc. (hatched areas) to indicate which classification result was obtained at the final node.

[0035] 1 is an example, and for example, a display based on the identification basis information may be displayed in various display formats such as a radar chart display or a graph display. Also, although an example in which the synthesis circuit 13 is provided in the explanation support device 1 has been described, the synthesis circuit 13 may also be built into the monitor MO. When the endoscopic image input to the explanation support device 1 is supplied from a processor that processes the signal of the image from the endoscope, the synthesis circuit 13 may be provided in this processor, or the synthesis circuit 13 may be provided as an independent device.

[0036] Next, the operation of the embodiment configured as above will be described with reference to Figures 3 and 4. Figure 3 is a flowchart for explaining the operation of the first embodiment, and Figure 4 is an explanatory diagram for explaining the operation of the first embodiment. In Figure 4, the same components as in Figure 1 are assigned the same reference numerals.

[0037] 4 shows an example in which an image of an animal (cat) is used as an inference target (identification target) image, and an inference is made as to what animal is shown as the identification result. The first inference unit IB is a black-box inference unit, and outputs an identification result that the image is a cat, and an inference result that the confidence level is 80%. The inference target image and the inference result of the first inference unit IB are supplied to a synthesis circuit 13.

[0038] On the other hand, the second inference unit IW1 and the third inference units IW2 and IW3 are white-box inference units that infer what animal the identification target is. These white-box inference units IW have different identification characteristics. In the example of FIG. 4 , the second inference unit IW1 is an inference unit that excels at inference using color information. The third inference unit IW2 is an inference unit that excels at inference using shape information, and the third inference unit IW3 is an inference unit that excels at inference using parts information.

[0039] The second reasoner IW1 outputs an inference result that the image is a fox with a certainty of 80% based on the identification grounds that the belly and legs are different colors. The third reasoner IW2 outputs an inference result that the image is a cat with a certainty of 50% based on the identification grounds that the ears are triangular. The third reasoner IW3 outputs an inference result that the image is a cat with a certainty of 75% based on the identification grounds that the distance between the eyes and nose is typical of cats.

[0040] The inference result of the first inference unit IB and each inference result (identification result and certainty factor) of each white-box inference unit IW are supplied to an identification unit 12a constituting the comparator 12. In S1 and S2 of Fig. 3, the identification unit 12a receives the first identification result and the first certainty factor from the first inference unit IB, and receives the second and third identification results and the second and third certainty factors from each white-box inference unit IW.

[0041] The identification unit 12a compares the first identification result with each of the second and third identification results (S3). As a result, the identification unit 12a determines which second and third identification results match the first identification result. In the example of FIG. 4 , the first identification result is “cat,” and the identification unit 12a determines that the identification results of the third inference units IW2 and IW3 match the first identification result. The identification unit 12a then identifies the confidence level closest to the first confidence level among the confidence levels associated with the second and third identification results that match the first identification result (S4). Since the first confidence level is 80%, the confidence level of the third inference unit IW2 is 50%, and the confidence level of the third inference unit IW3 is 75%, the identification unit 12a identifies the confidence level of the third inference unit IW3, 75%, as the confidence level closest to the first confidence level among the confidence levels associated with the second and third identification results that match the first identification result.

[0042] The explanation basis obtaining unit 12b obtains identification basis information linked to the certainty level determined by the determining unit 12a (S5). The identification basis information linked to the certainty level determined by the determining unit 12a is identification basis information from the third inference unit IW3, and the explanation basis obtaining unit 12b obtains the identification basis information, i.e., the information that "the distance between the eyes and ears is characteristic of cats," from the third inference unit IW3. The explanation basis obtaining unit 12b supplies the obtained identification basis information to the synthesis circuit 13.

[0043] The composition circuit 13 combines the image of the inference target, the inference result of the first inference unit IB, and an image based on the identification basis information from the comparator 12, and outputs display data for displaying the composite image to the monitor MO. In this way, the image of the inference target, the inference result of the first inference unit IB, and an image based on the identification basis information are compositely displayed on the display screen of the monitor MO (S6). In the example of Figure 4, the display screen of the monitor MO displays the image of a cat that is the identification target, a message saying "The image is a cat," which is an image based on the inference result of the first inference unit IB, and a message saying "Because the distance between the eyes and nose is typical of a cat," which is an image based on the identification basis information of the inference of the third inference unit IW3.

[0044] By viewing the display on the monitor MO, the user can obtain highly accurate inference results from the black-box type first inference unit IB, and can also obtain identification basis information from the white-box type inference unit IW. A plurality of inference units having different identification characteristics from the white-box type inference unit IW are used, and based on a comparison of the inference result of the first inference unit IB with the inference results of each white-box type inference unit IW, the white-box type inference unit IW that outputs an inference result that is considered to be more correct is selected to obtain the identification basis information, making it possible to obtain identification basis information that is suitable as an identification basis for the black-box type first inference unit IB.

[0045] In this manner, in this embodiment, a black-box type inference device and multiple white-box type inference devices are used, the identification result and certainty factor of the black-box type inference device are compared with each of the identification results and certainty factors of the multiple white-box type inference devices, and identification basis information is obtained from the white-box type inference device selected based on the comparison results, so that it is possible to obtain identification basis information for inference while obtaining highly accurate identification results.

[0046] (Modification) Fig. 5 is a block diagram showing a modification. In Fig. 5, the same components as those in Fig. 1 are assigned the same reference numerals, and their description will be omitted. In Fig. 1, each white-box reasoner IW has the same algorithm but is trained using different training data. In contrast, the white-box reasoner IW in Fig. 5 is composed of reasoners with different algorithms. For example, as the algorithm, a Bayesian network, a decision tree, a linear regression, a pattern matching, a support vector machine, a naive Bayes, k-NN, k-means, etc. can be adopted.

[0047] The modified example of Fig. 5 differs from Fig. 1 in that a second inference unit IW4 and third inference units IW5 and IW6 are used instead of the second inference unit IW1 and the third inference units IW2 and IW3, respectively. The second inference unit IW4 and the third inference units IW5 and IW6 are white-box inference units using a decision tree, linear regression, and a Bayesian network, respectively. Each of the second inference unit IW4 and the third inference units IW5 and IW6 performs classification on an input endoscopic image, outputs a classification result and its confidence level, and outputs classification basis information indicating the basis for each classification result. The second inference unit IW4 and the third inference units IW5 and IW6 are configured using mutually different algorithms and have mutually different classification characteristics.

[0048] Other configurations and effects are the same as those in FIG.

[0049] Second Embodiment FIG. 6 is a block diagram showing a second embodiment. In FIG. 6, the same components as those in FIG. 1 are denoted by the same reference numerals, and their description will be omitted. In the first embodiment, an example was shown in which a black-box inference device and multiple white-box inference devices are supplied with endoscopic images of the same frame acquired at the same time (same time axis). When observing a lesion with an endoscope, the endoscope and the lesion move relative to each other, so even if the same lesion is being observed, the image changes in different frames. For example, the position of the lesion in the image, the direction in which the lesion is observed, the size of the lesion, the focus level, color, brightness, etc., differ between different frames. Therefore, even if endoscopic images of the same lesion are obtained by capturing images of different frames, the classification results from each inference device may differ from each other. For this reason, it is possible that the classification results from a white-box inference device that performs inference on endoscopic images of different frames will be closer to the classification results from a black-box inference device than the classification results from a white-box inference device that performs inference on endoscopic images of the same frame, as shown in FIG. 1.

[0050] That is, in the first embodiment, multiple white-box reasoners with different discrimination characteristics were used to improve the possibility of obtaining an inference result close to that of the first inference unit IB, but in the second embodiment, a similar effect is expected by changing the image provided to the white-box reasoners.

[0051] In this embodiment, frame delayers DE1 and DE2 are provided to delay the endoscopic image from the image processing circuit 11. The frame delayer DE1 delays the endoscopic image from the image processing circuit 11 by, for example, one frame period and outputs it as a second image, while the frame delayer DE2 delays the endoscopic image from the image processing circuit 11 by a delay amount different from the delay amount of the frame delayer DE1, for example, two frame periods, and outputs it as the second image. Note that the delay amounts of the frame delayers DE1 and DE2 may be different from each other, and are not limited to one or two frame periods but may be set to three or more frame periods.

[0052] In this embodiment, second inference units IW11, IW12, and IW13 are employed instead of the second inference unit IW1 and the third inference units IW2 and IW3, respectively. The second inference units IW11, IW12, and IW13 are white-box inference units. The second inference unit IW11 performs classification on an input endoscopic image and outputs a classification result (second classification result) and its certainty (second certainty), as well as identification basis information indicating the basis for obtaining the classification result. Furthermore, the second inference units IW12 and IW13 each perform classification on an endoscopic image, which is the second image, and output a classification result (hereinafter referred to as a fourth classification result) and its certainty (hereinafter referred to as a fourth certainty), as well as identification basis information indicating the basis for obtaining each classification result.

[0053] In this embodiment, the second reasoners IW11, IW12, and IW13 (hereinafter referred to as the white-box reasoner IW10 when there is no need to distinguish between these reasoners) are reasoners trained using the same algorithm and the same training data, and have the same discrimination characteristics. Note that the white-box reasoners IW10 may have different discrimination characteristics from each other.

[0054] In the embodiment configured as described above, the endoscopic image from the image processing circuit 11 is supplied as is to the first inference unit IB and the second inference unit IW11, and is also delayed by a frame delay unit DE1 by, for example, one frame period before being supplied to the second inference unit IW12, and is also delayed by a frame delay unit DE2 by, for example, two frame periods before being supplied to the second inference unit IW13. The classification results and their certainties from each white-box inference unit IW10 are given to the comparator 12.

[0055] Even if the discrimination characteristics of the second inference units IW11, IW12, and IW13 are the same, since endoscopic images with different time axes are input, the discrimination results for the same lesion, their confidence levels, and discrimination basis information may differ between the white-box inference units IW10.

[0056] As in the first embodiment, the comparator 12 identifies the white-box reasoner IW10 among the second reasoners IW11, IW12, and IW13 that has obtained the same identification result as the first identification result of the first reasoner IB and has obtained a certainty level closer to the first certainty level, and obtains identification basis information from the identified white-box reasoner IW10 and outputs it to the synthesis circuit 13.

[0057] Other functions and effects are the same as those of the first embodiment.

[0058] 6 illustrates an example in which the second inference units IW11, IW12, and IW13 are used. However, a single white-box inference unit IW10 may be used, and a memory for storing the identification results, their certainty factors, and identification basis information may be provided. In this case, the output of the image processing circuit 11 and the outputs of the frame delay units DE1 and DE2 may be sequentially provided to the white-box inference unit IW10, or the frame delay unit may not be used and the output of the image processing circuit 11 may be provided to the white-box inference unit IW10 at predetermined time intervals. This allows multiple frames of different endoscopic images to be input to the white-box inference unit IW10, and the white-box inference unit IW10 may obtain the identification results, their certainty factors, and identification basis information in a time-division manner, which are then stored in memory. By providing the information stored in the memory to the comparator 12, an explanation support device similar to that shown in FIG. 6 can be constructed.

[0059] (Modification) In the embodiment of FIG. 6 , the white-box reasoner IW10 performs inference on images delayed by the frame delayers DE1 and DE2. In this case, each white-box reasoner IW10 may perform inference on a different lesion. Therefore, the control unit 10 may calculate the degree of match between the input endoscopic image and the delayed image from the frame delayers DE1 and DE2 for the input endoscopic image or a characteristic portion thereof, such as a lesion. The control unit 10, functioning as a match calculation unit, may provide the calculated degree of match to the comparator 12 and perform control so that the comparison target is a classification result based on an image in which the degree of match between the delayed image from the frame delayers DE1 and DE2 and the input endoscopic image is higher than a predetermined threshold. This enables comparison of valid classification results and confidence levels.

[0060] (Third Embodiment) Fig. 7 is a block diagram showing a third embodiment. In Fig. 7, the same components as those in Fig. 1 are given the same reference numerals and their explanations will be omitted. This embodiment employs a white-box type inference device that supports various preprocessing steps for adjusting image quality.

[0061] This embodiment differs from the embodiment of Figure 1 in that it employs a second inference unit IW21 and a third inference unit IW22, IW23 instead of the second inference unit IW1 and the third inference units IW2, IW3, and also adds a color enhancement unit 21, a contrast enhancement unit 22, and an outline enhancement unit 23.

[0062] The second inference unit IW21 and the third inference units IW22 and IW23 (hereinafter, when there is no need to distinguish between these inference units, they will be referred to as the white-box inference unit IW20) are white-box inference units. Each of the second inference unit IW21 and the third inference units IW22 and IW23 performs classification on an input endoscopic image, outputs a classification result and its certainty, and outputs classification basis information indicating the basis for each classification result. In this embodiment, the second inference unit IW21 is trained using color-enhanced endoscopic images as training data. Meanwhile, the third inference units IW22 and IW23 are trained using contrast-enhanced or contour-enhanced endoscopic images as training data. That is, the second inference unit IW21, the third inference unit IW22, and the third inference unit IW23 have mutually different classification characteristics.

[0063] In FIG. 7, for convenience, the white-box type inference device is shown divided into a second inference device IW21 and two third inference devices IW22 and IW23. However, this does not mean that the second inference device IW21 and the third inference devices IW22 and IW23 are different types, and the number of third inference devices can be set to one or more as appropriate.

[0064] The color enhancement unit 21 performs color enhancement processing on the endoscopic image from the image processing circuit 11 and then provides the image to the second inference unit IW21. The contrast enhancement unit 22 performs contrast enhancement processing on the endoscopic image from the image processing circuit 11 and then provides the image to the third inference unit IW22. The contour enhancement unit 23 performs contour enhancement processing on the endoscopic image from the image processing circuit 11 and then provides the image to the third inference unit IW23.

[0065] In the embodiment configured as described above, the endoscopic image from the image processing circuit 11 is color-enhanced by the color enhancement unit 21 and then provided to the second inference unit IW21, contrast-enhanced by the contrast enhancement unit 22 and then provided to the third inference unit IW22, and edge-enhanced by the edge enhancement unit 23 and then provided to the third inference unit IW23. The second inference unit IW21, the third inference unit IW22, and IW23 each have high accuracy in classifying color-enhanced images, contrast-enhanced images, and edge-enhanced images. As a result, each white-box inference unit IW20 can obtain a classification result, its certainty, and classification basis information with relatively high accuracy. The classification result, its certainty, and classification basis information from each white-box inference unit IW20 are supplied to the synthesis circuit 13.

[0066] The other configurations and effects are the same as those of the embodiment shown in FIG.

[0067] (Fourth embodiment) Fig. 8 is a block diagram showing a fourth embodiment. In Fig. 8, the same components as those in Fig. 1 are given the same reference numerals and their description will be omitted. This embodiment employs a white-box inference device that supports various types of image processing for endoscopic images.

[0068] This embodiment differs from the embodiment of Figure 1 in that it employs second inference units IW31, IW32, and IW33 instead of the second inference unit IW1 and the third inference units IW2 and IW3, respectively, and also provides a user correction unit 31, in which an endoscopic image input to the image processing circuit 11 is provided to the second inference unit IW31, an endoscopic image output from the image processing circuit 11 is provided to the first inference unit IB and the second inference unit IW32, and the output of the user correction unit 31 is provided to the second inference unit IW33.

[0069] The user correction unit 31 applies various image processing to the endoscopic image (first image) output from the image processing circuit 11, such as adding at least one of color, contrast, sharpness, inversion, rotation, reduction, enlargement, distortion, and partial cropping, to obtain and output a third image.

[0070] The second inference units IW31, IW32, and IW33 (hereinafter, when there is no need to distinguish between these inference units, they will be referred to as the white-box inference unit IW30) are white-box inference units. The second inference unit IW31 performs classification on an input endoscopic image, outputs the classification result and its confidence level, and outputs classification basis information indicating the basis for each classification result. Furthermore, each of the second inference units IW32 and IW33 performs classification on a third image obtained by performing image processing on the input endoscopic image based on image processing by the image processing circuit 11 or image processing by the user correction unit 31, and outputs the classification result (hereinafter, referred to as the fifth classification result) and its confidence level (hereinafter, referred to as the fifth confidence level), as well as outputting identification basis information indicating the basis for each classification result.

[0071] (identification characteristics are the same) Fig. 9 is an explanatory diagram for explaining the embodiment of Fig. 6 and Fig. 8. In Fig. 9, the same components as those in Fig. 6 and Fig. 8 are given the same reference numerals.

[0072] As described above, in order to obtain high inference accuracy using a white-box inference device, the first and second embodiments provide multiple white-box inference devices IW with different classification results. In the second embodiment of FIG. 6, different images with different time axes are input to each white-box inference device IW10, and in the fourth embodiment of FIG. 8, different images with different image processing are input to each white-box inference device IW30. Therefore, even if the classification characteristics of the white-box inference devices IW10 are the same, or even if the classification characteristics of the white-box inference devices IW30 are the same, the classification results and certainty factors supplied to the comparator 12 are different. Therefore, the same classification characteristics may be set for each white-box inference device IW30, as with the white-box inference device IW10.

[0073] 9 shows this state using the same notation as in FIG. 4. In the example of FIG. 9, the cat image output from the image processing circuit 11 is supplied to the first inference unit IB and the second inference unit IW11 or IW31. A second image obtained by processing the cat image from the image processing circuit 11 into N1 frames and a third image obtained by image-processing the cat image from the image processing circuit 11 are supplied to the second inference units IW12 and IW32. Furthermore, a second image obtained by processing the cat image from the image processing circuit 11 into N2 frames and a third image obtained by further image processing the cat image from the image processing circuit 11 are supplied to the second inference units IW13 and IW33. Therefore, even if the discrimination characteristics of the white-box inference units IW10 are identical to each other or the discrimination characteristics of the white-box inference units IW30 are identical to each other, different discrimination results and certainties can be obtained between the white-box inference units IW10 or IW30, for example, as in FIG. 4.

[0074] Therefore, in the fourth embodiment, the same effects as in the first embodiment can be obtained.

[0075] (Different Discrimination Characteristics) In the embodiment of FIG. 8 , the second inference units IW31, IW32, and IW33 may be set to have different discrimination characteristics. For example, the second inference unit IW31 may be constructed by learning using, as training data, endoscopic images before image processing by the image processing circuit 11, such as RAW images. Furthermore, the second inference units IW32 and IW33 may be constructed by learning using, as training data, images after image processing by the image processing circuit 11 or endoscopic images after image processing by the user correction unit 31. That is, in this case, the second inference units IW31, IW32, and IW33 have different discrimination characteristics, and each has a characteristic of high discrimination accuracy for images before image processing by the image processing circuit 11, images after image processing by the image processing circuit 11, and images after image processing by the user correction unit 31.

[0076] In this case, the endoscopic image before image processing by the image processing circuit 11 is provided to a second inference unit IW31, the image after image processing by the image processing circuit 11 is provided to a second inference unit IW32, and the image after image processing by the user correction unit 31 is provided to a second inference unit IW33. The second inference units IW31, IW32, and IW33 each obtain a classification result, its certainty factor, and classification basis information with relatively high accuracy.

[0077] The other configurations and effects are the same as those of the embodiment shown in FIG.

[0078] Fifth Embodiment Fig. 10 is a block diagram showing a fifth embodiment. In Fig. 10, the same components as those in Fig. 1 are designated by the same reference numerals, and a description thereof will be omitted. This embodiment is an example having a plurality of groups of white-box reasoners (hereinafter referred to as "inference unit groups"), each group including the second inference unit IW1 and the third inference units IW2 and IW3 of Fig. 1. Note that each inference unit group may include two or more white-box reasoners.

[0079] In Fig. 10, an example is shown in which inference unit groups IWG1 to IWG3 each including the second inference unit IW1 and the third inference units IW2 and IW3 are provided instead of the second inference unit IW1 and the third inference units IW2 and IW3 in Fig. 1. Also, in Fig. 10, an information unit 9 is added to the configuration in Fig. 1.

[0080] In CAD, a white-box inference device may be used for each part of a living body. Each inference device group in FIG. 10 may have an inference device corresponding to each part of a living body.

[0081] The information unit 9 acquires various types of information from the outside and image information from the image processing circuit 11. The control unit 10, which serves as a second white-box inference unit control unit, selects an inference unit group IWG to be driven from among IWG1 to IWG3 (hereinafter, when it is not necessary to distinguish between these inference unit groups, they will be referred to as inference unit groups IWG) based on the various types of information acquired by the information unit 9. That is, the information unit 9 acquires information regarding the selection conditions used by the control unit 10 to make the selection. For example, the information unit 9, which serves as a site information unit, can receive site identification information indicating which site in the living body the endoscopic image is an image of and output it to the control unit 10. The control unit 10 selects an inference unit IWG to be driven from among the inference unit IWG based on the selection conditions. Note that the site identification information may be generated based on a user operation on an operation unit (not shown).

[0082] The information unit 9 may be configured with a processor using a CPU, FPGA, or the like, and may acquire information by operating according to a program stored in a memory (not shown), or by implementing some or all of its functions in a hardware electronic circuit. The information unit 9 may receive an endoscopic image from the image processing circuit 11, analyze the endoscopic image, and determine which part of the human body the image portion in the endoscopic image represents, and provide the determination result to the control unit 10. For example, the information unit 9 may refer to a database (not shown) that records the characteristic color and shape of each part of the body, as well as patterns of blood vessels visible on the surface, and compare the image characteristics obtained by image analysis to determine the part of the body. The information unit 9 may also use an organ inference device (not shown) that determines the part of the human body the endoscopic image represents. The information unit 9 may also determine the part of the body based on information acquired by a method in which an external receiver receives transmissions from a transmitter (such as a magnet) at the tip of the endoscope and determines the position of the tip of the endoscope on the body. The information unit 9 may also determine the site based on the detection result of the insertion length of the scope.

[0083] The inference unit groups IWG1 to IWG3 correspond to, for example, various parts of the body. Each inference unit group IWG is composed of a second inference unit IW1 and multiple third inference units IW2 and IW3, as in Fig. 1, and the second inference unit IW1 and the third inference units IW2 and IW3 correspond to, for example, different parts of the body for each inference unit group IWG. Note that, although Fig. 10 shows three inference unit groups IWG, the number of inference unit groups IWG can be set as appropriate.

[0084] The control unit 10 is provided with part identification information from the information unit 9, and is configured to provide a selection signal to the corresponding inference unit group IWG to select the inference unit group IWG corresponding to the part. Of the inference unit groups IWG, the inference unit group IWG that receives the selection signal becomes operable, and the second inference unit IW1 and the third inference units IW2 and IW3 included in the operable inference unit group IWG supply the identification result, its certainty factor, and identification environment information to the comparator 12.

[0085] Other configurations and effects are the same as those of the first embodiment. It is clear that this embodiment can also be applied to the second embodiment.

[0086] (Selection for Each Type of Information) In the above description, an example has been described in which the control unit 10 determines the inference unit group IWG to select according to the part of the human body, but the inference unit group IWG to be driven may be selected from each inference unit group IWG based on predetermined selection conditions acquired by the information unit 9, regardless of the part. Note that the information unit 9 can acquire various information based on input operations made by the user to the operation unit, and can also acquire information for obtaining selection conditions from endoscopic images from the image processing circuit 11, and can also acquire information for obtaining selection conditions from a light source device that drives the endoscope, etc.

[0087] For example, the information unit 9 as an optical information unit can receive optical information indicating the type of observation light (illumination light) used when capturing images with an endoscope (not shown) and output it to the control unit 10. Each inference unit group IWG includes one that includes a white-box type inference unit with high inference accuracy for captured images obtained using white light as the observation light, one that includes a white-box type inference unit with high inference accuracy for captured images obtained using NBI (narrow band illumination) light as the observation light, one that includes a white-box type inference unit with high inference accuracy for captured images obtained using MBI light as the observation light, and the like.

[0088] In this case, the control unit 10 selects and drives the inference unit group IWG corresponding to the type of observation light used when capturing the endoscopic image, based on the light information from the information unit 9. This makes it possible to obtain a more accurate identification result, certainty factor, and region identification information according to the observation light.

[0089] Furthermore, for example, the information unit 9 serving as an observation mode information unit can receive observation mode information indicating the type of observation mode of the endoscope and output it to the control unit 10. Each inference unit group IWG includes an inference unit corresponding to the observation mode, such as one including a white-box inference unit with high inference accuracy for endoscopic images acquired in a normal observation mode, or one including a white-box inference unit with high inference accuracy for endoscopic images acquired in a special light observation mode such as NBI.

[0090] In this case, the control unit 10 selects and drives the inference unit group IWG corresponding to the type of observation mode of the endoscopic image based on the observation mode information from the information unit 9. This makes it possible to obtain more accurate identification results, confidence levels, and site identification information according to the observation mode.

[0091] Furthermore, for example, the information unit 9 serving as a model information unit can receive model information indicating the model of an endoscope (not shown) and output it to the control unit 10. Each inference unit group IWG is based on the type of endoscope, for example, one that includes a white-box type inference unit with high inference accuracy for endoscopic images obtained using an upper gastrointestinal endoscope, one that includes a white-box type inference unit with high inference accuracy for endoscopic images obtained using a lower gastrointestinal endoscope, etc.

[0092] In this case, the control unit 10 selects and drives the inference unit group IWG corresponding to the endoscope model based on the model information from the information unit 9. This makes it possible to obtain more accurate identification results, confidence levels, and site identification information according to the endoscope model.

[0093] Furthermore, for example, the information unit 9 can receive information indicating whether the endoscopic image is a magnified image or a non-magnified image, and output the information to the control unit 10. When the endoscopic image is magnified, it is easy to observe the vascular pattern. For example, each inference unit group IWG may include one that includes a white-box inference unit with high inference accuracy for endoscopic images including vascular patterns, or one that includes a white-box inference unit with high inference accuracy for non-magnified images.

[0094] In this case, the control unit 10 selects and drives the inference unit group IWG depending on whether the endoscopic image is a magnified image or a non-magnified image, based on information from the information unit 9. This makes it possible to obtain a more highly accurate identification result, a degree of certainty, and region identification information depending on whether the endoscopic image is a magnified image or not.

[0095] Furthermore, the information unit 9 may have a memory (not shown) that stores, for each white-box inference unit IW of each inference unit group IWG, the number of times or the rate at which the same identification result as the first identification result is output (selection rate). For example, the information unit 9 may provide information to the control unit 10 for selecting an inference unit group IWG that includes many white-box inference units IW with high selection rates, so that the same identification result as the first identification result is more likely to be obtained, based on the information on the selection rate. Note that, when acquiring the selection rate, the information unit 9 may determine that an identification result that is within ±10% of the first identification result is the same as the first identification result.

[0096] Furthermore, for example, the information unit 9 may acquire information on the lesion discrimination result obtained by the first inference unit IB and supply the acquired information to the control unit 10. Each inference unit group IWG includes a white-box inference unit with high inference accuracy depending on the type of lesion. For example, if the discrimination result acquired by the information unit 9 is "gastric cancer," the control unit 10 selects and drives an inference unit group IWG including a white-box inference unit IW with high inference accuracy for pattern classification of stomach diseases and stomach cancer. This makes it possible to obtain a more accurate identification result, confidence level, and site identification information depending on the disease.

[0097] (Sixth embodiment) Fig. 11 is a block diagram showing a sixth embodiment. In Fig. 11, the same components as those in Fig. 1 are given the same reference numerals and their explanations will be omitted. This embodiment is an example in which none of the white-box reasoners are used, as in the embodiment in Fig. 10.

[0098] This embodiment differs from the embodiment of FIG. 1 in that a priority storage unit 14 is employed and the control unit 10 selectively operates the white-box reasoner IW according to the contents stored in the priority storage unit 14.

[0099] The white-box reasoner IW that outputs the certainty level identified by the identification unit 12a is referred to as the white-box reasoner associated with the identified certainty level. That is, this white-box reasoner refers to the white-box reasoner IW that outputs the certainty level (certainty level identified by the identification unit 12a) having a value closest to the first certainty level among the certainty levels output from the white-box reasoner IW whose identification result matches the first identification result (certainty levels associated with the same identification result as the first identification result). The priority storage unit 14 stores information related to the priorities of the white-box reasoners IW so that the priority of the white-box reasoner IW associated with the identified certainty level is higher than the priorities of other white-box reasoners IW (hereinafter referred to as white-box reasoners not associated with the identified certainty level).

[0100] The control unit 10, which serves as a first white-box inference unit control unit, supplies each white-box inference unit IW with a selection signal for driving the white-box inference units IW in descending order of priority based on the priority information stored in the priority storage unit 14 (not shown). The white-box inference unit IW that outputs the certainty factor identified by the identification unit 12a is considered to be an inference unit with good performance, such as a high matching rate of the identification results. For example, when the number of white-box inference units IW is sufficiently large, the control unit 10 may select and operate approximately one to three white-box inference units IW in descending order of priority using the priority information stored in the priority storage unit 14 based on a prior evaluation. Limiting the number of white-box inference units to be operated has the effect of enabling highly accurate inference while reducing power consumption.

[0101] Furthermore, the control unit 10 may be configured to acquire from the comparator 12a information on whether a certainty level, which is a difference from the first certainty level within a predetermined range, has been identified among the certainty levels associated with the same identification result as the first identification result (not shown). When a certainty level, which is a difference from the first certainty level within a predetermined range, has been identified, the control unit 10 may control each white-box reasoner IW (not shown) so as not to start driving the white-box reasoner IW thereafter, even if there is a white-box reasoner IW with a relatively high priority that was scheduled to be driven. In this case, the explanation basis acquiring unit 12b may be controlled by the control unit 10 (not shown) to acquire identification basis information (hereinafter referred to as identification basis information associated with a certainty level, the difference from the first certainty level being within a predetermined range) obtained from the white-box reasoner IW that outputs a certainty level determined to be a difference from the first certainty level within a predetermined range, and output the information to the monitor MO.

[0102] The control unit 10 may be configured to store, for each white-box reasoner IW, the number of times or the rate at which the same identification result as the first identification result is output (selection rate) in the priority storage unit 14. In this case, the control unit 10 may be configured to provide a selection signal to the white-box reasoner IW for selecting a white-box reasoner IW with a higher selection rate, so that the same identification result as the first identification result is more likely to be output, based on the information on the selection rate stored in the priority storage unit 14. When acquiring the selection rate, the control unit 10 may determine that an identification result within ±10% of the first identification result is the same as the first identification result. In other words, the control unit 10 may select an inference unit to be driven from among the white-box reasoners IW based on the first identification result as a predetermined condition.

[0103] (Seventh embodiment) Fig. 12 is a block diagram showing a seventh embodiment. In Fig. 12, the same components as those in Fig. 1 are given the same reference numerals and their explanations will be omitted. This embodiment is configured with a separate inference unit for lesion detection, which detects the lesion position (area), and a separate inference unit for differentiation (disease name detection).

[0104] This embodiment differs from the embodiment of Figure 1 in that a first inference unit IB41 is used instead of the first inference unit IB, a second inference unit IW41 and a third inference unit IW42 and IW43 are used instead of the second inference unit IW1 and the third inference units IW2 and IW3, respectively, and a first inference unit IB42 for lesion detection is provided.

[0105] The first inference unit IB42 for lesion detection is configured as a black-box inference unit. The first inference unit IB42 for lesion detection may be constructed, for example, by deep learning using a multi-layer neural network. The first inference unit IB42 for lesion detection identifies the position (area) of a lesion or the like contained in an input endoscopic image and obtains information for generating a frame image indicating the identified area. The information on the lesion area from the first inference unit IB42 for lesion detection is supplied to the first inference unit IB41.

[0106] The first inference unit IB41 is configured as a black-box inference unit. The first inference unit IB41 may be constructed, for example, by deep learning using a multi-layer neural network. The first inference unit IB41 performs discrimination on the image of the lesion area indicated by the first inference unit for lesion detection IB42 and obtains discrimination information indicating the discrimination result. The discrimination result (first discrimination result) including the frame image and discrimination information obtained by the first inference unit for lesion detection IB42 and the first inference unit IB41, and the certainty factor of the first discrimination result (first certainty factor) are supplied to the synthesis circuit 13 as lesion information.

[0107] The second inference unit IW41 and the third inference units IW42 and IW43 (hereinafter, referred to as the white-box inference unit IW40 when there is no need to distinguish between these inference units) are white-box inference units. These second inference unit IW41 and the third inference units IW42 and IW43 are provided with information on lesion areas from the first lesion detection inference unit IB42. The white-box inference units IW40, like the white-box inference unit IW in FIG. 1, have different discrimination characteristics and perform discrimination on images of lesion areas. Each white-box inference unit IW40 obtains a discrimination result (discrimination result), its certainty factor, and discrimination basis information. The comparator 12 obtains the discrimination basis information by comparing the discrimination result and certainty factor of the first inference unit IB41 with the discrimination result and certainty factor of each white-box inference unit IW40. That is, the explanation basis acquisition unit 12b acquires identification basis information from the white-box reasoning device IW40, which outputs the certainty factor (certainty factor identified by the identification unit 12a) having the value closest to the first certainty factor among the certainty factors associated with the same identification result as the first identification result.

[0108] The other configurations and effects are the same as those of the first embodiment.

[0109] In each of the above embodiments, a threshold may be determined, and the white-box reasoners may be operated in order until a white-box reasoner with a certainty level exceeding the threshold is found. In this case, the overall processing time may be lengthened and a white-box reasoner with a certainty level close to the first certainty level may not be detected, but the effect is that the load on the CPU can be reduced. Also, in each of the above embodiments, the control unit 10 may selectively operate a white-box reasoner or a group of reasoners, but instead of controlling these operations, the control unit 10 may control the comparison in the comparator 12 so that the identification results from non-selected reasoners are not used for comparison.

[0110] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0111] Furthermore, among the technologies described herein, many of the controls and functions, mainly those described in the flowcharts, can be set by a program, and the above-mentioned controls and functions can be realized by having a computer read and execute the program. The program can be recorded or stored, in whole or in part, as a computer program product on portable media such as flexible disks, CD-ROMs, non-volatile memories, or storage media such as hard disks and volatile memories, and can be distributed or provided at the time of product shipment, via portable media, or via communication lines. A user can easily realize the explanation assistance device of this embodiment by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.

Claims

1. An explanation assistance device comprising: an identification unit that receives a first classification result and a first certainty factor output by a first black-box type inference device for a first image, and a second classification result and a second certainty factor output by a second white-box type inference device for the first image; further receives at least one of a third classification result and a third certainty factor output by a third white-box type inference device different from the second inference device for the first image, a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image, and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image; and an explanation basis acquisition unit that acquires identification basis information linked to the identified certainty factor and outputs it to a monitor.

2. The explanation support device according to claim 1, wherein when the third identification result and the third certainty are received, the second inference device and the third inference device have the same algorithm but different machine learning data.

3. The explanation support device according to claim 1, wherein when the third identification result and the third certainty factor are received, the second reasoner and the third reasoner have different algorithms.

4. The explanation support device according to claim 1, wherein, when the second inference device and the third inference device have different algorithms, the second inference device and the third inference device are selected from Bayesian networks, decision trees, linear regression, pattern matching, support vector machines, naive Bayes, k-NN, and k-means so as to be of different types from each other.

5. The explanation support device of claim 1, wherein when the fifth identification result and the fifth certainty level are received, the third image is an image to which at least one of the following processing has been applied to the first image: color, contrast, sharpness, inversion, rotation, reduction, enlargement, distortion, and partial cropping.

6. An explanation support device as described in claim 1, comprising: a priority memory unit that, when receiving a third identification result and a third certainty, stores the priority of the inference unit associated with the identified certainty out of the second inference unit and the third inference unit so that it is higher than the priority of the inference unit not associated with the identified certainty; and a first white-box inference unit control unit that controls the driving of the second inference unit and the third inference unit, wherein the first white-box inference unit control unit drives the second inference unit and the third inference unit in descending order of priority based on the stored priorities.

7. The explanation support device described in claim 6, wherein, when the second inference device and the third inference device are driven in descending order of priority based on priority, if the identification unit identifies a certainty level among the certainty levels associated with the same identification result as the first identification result, the difference from the first certainty level is within a predetermined range, the first white-box inference device control unit does not start driving the next white-box inference device in the order even if there is a white-box inference device that is not being driven, and the explanation basis acquisition unit acquires identification basis information linked to a certainty level whose difference from the first certainty level is within the predetermined range and outputs it to a monitor.

8. An explanation support device as described in claim 1, comprising two or more groups of inference devices, each of which includes at least the second inference device and the third inference device for said group of inference devices, and comprising at least a second white-box inference device control unit that controls the driving of the second inference device and the third inference device, and the second white-box inference device control unit selects a group of inference devices to be driven from among said groups of inference devices based on predetermined conditions.

9. An explanation support device as described in claim 8, further comprising a part information unit that receives part identification information that identifies the part in the living body from which the first image was captured, and the second white-box reasoner control unit selects, as the predetermined condition, a group of reasoners to be driven from among the two or more types of inference unit groups based on the part identification information.

10. An explanation support device as described in claim 8, including an optical information unit that receives optical information that identifies the observation light when the first image was captured, and the second white-box reasoner control unit selects, as the specified condition, a group of reasoners to be driven from among the two or more types of inference unit groups based on the optical information.

11. An explanation support device as described in claim 8, including a model information unit that receives model information of the endoscope that captured the first image, and the second white-box reasoner control unit selects, as the predetermined condition, a group of reasoners to be driven from among the two or more types of inference unit groups based on the model information.

12. An explanation support device as described in claim 8, further comprising an observation mode information unit that receives observation mode information of the endoscope that captured the first image, and wherein the second white-box reasoner control unit selects, as the predetermined condition, a group of reasoners to be driven from among the two or more types of inference unit groups based on the observation mode information.

13. The explanation support device according to claim 8, wherein the second white-box reasoner control unit selects a group of reasoners to be driven from the two or more types of reasoner groups based on the first identification result as the predetermined condition.

14. An explanation support device including a processor, which receives a first classification result and a first confidence level for a first image output by a first black-box inference device, and a second classification result and a second confidence level for the first image output by a second white-box inference device; further receives at least one of a third classification result and a third confidence level for the first image output by a third white-box inference device different from the second inference device; a fourth classification result and a fourth confidence level output by the second inference device for a second image that is a predetermined number of frames away from the first image; and a fifth classification result and a fifth confidence level output by the second inference device for a third image obtained by processing the first image; identifies, from the received confidence levels other than the first confidence level, a confidence level closest to the first confidence level that is associated with the same classification result as the first classification result; and obtains identification basis information associated with the identified confidence level and outputs it to a monitor.

15. A method of supporting explanations, comprising: receiving a first classification result and a first confidence level for a first image output by a first black-box inference device, and a second classification result and a second confidence level for the first image output by a second white-box inference device; further receiving at least one classification result and confidence level from among a third classification result and a third confidence level for the first image output by a third white-box inference device different from the second inference device; a fourth classification result and a fourth confidence level for a second image that is a predetermined number of frames away from the first image output by the second inference device; and a fifth classification result and a fifth confidence level for a third image obtained by processing the first image; identifying, from among the confidence levels other than the first confidence level received, a confidence level closest to the first confidence level that is associated with the same classification result as the first classification result; and obtaining identification basis information associated with the identified confidence level and outputting it to a monitor.

16. A program for supporting explanations that causes a computer to execute the following procedures: receive a first classification result and a first certainty factor output by a first black-box inference device for a first image, and a second classification result and a second certainty factor output by a second white-box inference device for the first image; further receive at least one classification result and certainty factor from among a third classification result and a third certainty factor output by a third white-box inference device different from the second inference device for the first image, a fourth classification result and a fourth certainty factor output by the second inference device for a second image that is a predetermined number of frames away from the first image, and a fifth classification result and a fifth certainty factor output by the second inference device for a third image obtained by processing the first image; identify, from among the certainty factors other than the first certainty factor received, the certainty factor closest to the first certainty factor that is associated with the same classification result as the first classification result; and obtain identification basis information associated with the identified certainty factor and output it to a monitor.

17. A non-transitory recording medium having recorded thereon an explanation support program that executes the following steps: receiving a first classification result and a first confidence level output by a first black-box inference device for a first image, and a second classification result and a second confidence level output by a second white-box inference device for the first image; further receiving at least one of a third classification result and a third confidence level output by a third white-box inference device different from the second inference device for the first image, a fourth classification result and a fourth confidence level output by the second inference device for a second image that is a predetermined number of frames away from the first image, and a fifth classification result and a fifth confidence level output by the second inference device for a third image obtained by processing the first image; identifying, from among the confidence levels other than the first confidence level received, the confidence level closest to the first confidence level that is associated with the same classification result as the first classification result; and obtaining identification grounds information associated with the identified confidence level and outputting it to a monitor.

Images