Diagnostic support devices, diagnostic support methods, diagnostic support programs

Abstract

The goal is to generate more accurate diagnostic results based on text and image information related to the examination. [Solution] The diagnostic support device of the present invention comprises an input unit, a feature extraction unit, a filter unit, and a diagnostic assistance unit. The input unit acquires pairs of text information and image information related to the examination. The feature extraction unit outputs feature information, which is information indicating findings on the image information and the likelihood of said findings. The filter unit outputs the findings indicated by the feature information as corrected feature information when the likelihood of the findings indicated by the feature information exceeds a threshold for the image features included in said findings. Alternatively, the filter unit calculates a weighted likelihood using the likelihood of the findings indicated by the feature information and the weights for the image features included in said findings, and outputs the feature information selected based on the weighted likelihood as corrected feature information. The diagnostic assistance unit takes text information and corrected feature information as input and outputs a proposed diagnostic result.

Description

Technical Field

[0001] The present invention relates to a diagnostic support device, a diagnostic support method, and a diagnostic support program that output a diagnostic result proposal based on text information and image information related to an examination.

Background Art

[0002] As conventional techniques for analyzing multimodal data such as images and texts using artificial intelligence (AI), Patent Documents 1 and 2 are known. Patent Documents 1 and 2 show examples in which a model for extracting features of an image and the like are combined with a multimodal model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to create a highly accurate learned model, a huge number of learning processes are required. For a learned model premised on general-purpose use, a huge number of learning processes are possible, and technology is advancing rapidly. However, when seeking an output in a text format related to a limited field, it is difficult to collect learning data, and it is difficult to improve the accuracy of the learned model. Therefore, it is difficult to construct a learning model that outputs a diagnostic result proposal in a text format from text information and image information related to an examination. The object of the present invention is to be able to output a more accurate diagnostic result proposal from text information and image information related to an examination. )

Means for Solving the Problems

[0005] The diagnostic support device of the present invention comprises a recording unit, an input unit, a feature extraction unit, a filter unit, and a diagnostic support unit. The recording unit records threshold values ​​or weights for each of a plurality of predetermined image features. The input unit acquires pairs of text information and image information related to the examination. The feature extraction unit outputs feature information, which is information indicating findings for the image information and the likelihood of those findings. The filter unit outputs the feature information as modified feature information when the likelihood of the findings indicated by the feature information exceeds the threshold value for the image features included in the findings. Alternatively, the filter unit calculates a weighted likelihood using the likelihood of the findings indicated by the feature information and the weights for the image features included in the findings, and outputs the feature information selected based on the weighted likelihood as modified feature information. The diagnostic support unit takes text information and modified feature information as input and outputs a proposed diagnostic result. [Effects of the Invention]

[0006] According to the diagnostic support device of the present invention, the filter unit processes the features obtained from the image information, allowing appropriate findings to be input to the diagnostic support unit. Therefore, a more accurate proposed diagnostic result can be output. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing an example of the functional configuration of the diagnostic support device in Example 1. [Figure 2] A diagram showing an example of the processing flow of the diagnostic support device in Example 1. [Figure 3] A diagram showing an example of the functional configuration of the diagnostic support device in Modification Example 1. [Figure 4] This figure shows an example of the processing flow of the diagnostic support device in Modification Example 1. [Figure 5] This figure shows an example of the functional configuration of a diagnostic support device in modified example 2. [Figure 6] This diagram shows an example of the processing flow of a diagnostic support device in modified example 2. [Figure 7] A diagram illustrating an example of a computer's functional configuration. [Modes for carrying out the invention]

[0008] The embodiments of the present invention will be described in detail below. Components having the same function will be given the same number, and redundant explanations will be omitted. [Examples]

[0009] Figure 1 shows an example of the functional configuration of the diagnostic support device of Example 1, and Figure 2 shows an example of the processing flow of the diagnostic support device of Example 1. The diagnostic support device 100 includes a recording unit 190, an input unit 110, a feature extraction unit 120, a filter unit 130, and a diagnostic assistance unit 140, and outputs a proposed diagnostic result based on text information and image information related to the examination. "Text information and image information related to the examination" refers to, for example, prior information (text information) shown in text in health checkups, detailed examinations of the human body, etc., and image information such as chest X-ray images and chest CT (Computed Tomography) images. However, it is not limited to the chest, and other parts such as the head may also be used, and the type of examination may be other than X-ray images and CT, such as MRI (Magnetic Resonance Imaging), SPECT (Single Photon Emission Computed Tomography), PET (Positron Emission Tomography), ultrasound images, etc.

[0010] The recording unit 190 records threshold values ​​for each of several predetermined image features. Image features include words such as "normal" (no abnormality), "normal, postoperative change" (abnormality present but after treatment), "abnormal, calcification" (high probability of post-treatment change), "abnormal, nodular shadow," "abnormal, mass shadow," "abnormal, infiltrative shadow," "abnormal, linear shadow," and "abnormal, granular shadow." If two chest X-ray images taken on different dates are input as image information, it is also possible to use changes in "ground-glass opacity" as an image feature. Threshold values ​​should be set for each word that describes an image feature.

[0011] The input unit 110 acquires pairs of text information and image information related to the examination (S110). The "text information" may include information such as gender and age, as well as information such as the date of the examination, reason for the examination, body temperature, blood test results, and history of past illnesses and treatments. The "image information" may be a single chest X-ray, a combination of a frontal chest X-ray and a lateral chest X-ray, a combination of chest X-rays from different dates, a combination of multiple images such as CT images, or a combination of images from different types of examinations such as a chest X-ray and a CT image. Note that the text information acquired by the input unit 110 also includes cases where there is no text information.

[0012] The feature extraction unit 120 outputs feature information, which is information indicating findings for the image information and the likelihood of those findings (S120). "Findings" are information indicating the features and location of the image obtained from the image information. The image features obtained from the image information may be the words that indicate the image features as described above, or they may be symbols or numbers that identify the words that indicate the image features. The location information included in the findings may be, for example, XY coordinates or combinations of XY coordinates. "Likelihood of findings" means the probability (likelihood) that the image at the location indicated by the location information is the image feature (the word mentioned above). The feature extraction unit 120 may include a model such as a convolutional neural network (CNN). The model such as the convolutional neural network of the feature extraction unit 120 should be trained using combinations of image information (input) and correct findings (output) limited to the field in which the diagnostic support device 100 is used as training data. The feature extraction unit 120 does not need to output only one finding; it may output multiple findings along with their likelihoods. Furthermore, the feature extraction unit 120 may also output tensors, which are intermediate products of a convolutional neural network, as feature information. As mentioned above, "intermediate products" refer to tensor-formatted data from any layer within a model such as a convolutional neural network.

[0013] The filter unit 130 outputs the feature information as corrected feature information when the likelihood of the finding indicated by the feature information exceeds a threshold for the image features included in the finding (S130). The output of the model, such as a convolutional neural network, of the feature extraction unit 120 depends on the training data. In reality, even image information and findings that occur very infrequently need to be included in the training data. And, in order to train the model, the proportion of image information and findings that occur very infrequently in the training data may have to be higher than the actual occurrence frequency. For example, when such training is performed, there is a possibility that findings that actually occur very infrequently will be output by the feature extraction unit 120 at a higher frequency than they actually are. Therefore, by setting a low threshold for features that occur frequently and a high threshold for features that occur infrequently, the frequency at which findings are output will be brought closer to the actual occurrence frequency, and appropriate findings can be output as corrected feature information. Also, if the feature extraction unit 120 also outputs intermediate products, the diagnostic support unit 140 should add the intermediate products to the corrected feature information when intermediate products are also necessary for the findings. Furthermore, any findings that use the intermediate product as modified characteristic information should also be recorded in the recording unit 190.

[0014] The diagnostic support unit 140 takes text information and modified feature information as input and outputs a proposed diagnostic result (S140). The input to the diagnostic support unit 140 may be the text information itself acquired by the input unit 110 and the modified feature information itself output by the filter unit 130, or it may be information that has been processed, such as by changing the format. For example, symbols or numbers that identify words that represent image features may be converted into words that represent image features before being input. Image information may also be included as input to the diagnostic support unit 140.

[0015] The diagnostic assistance unit 140 may include large language models (LLMs) and large multimodal models. The model of the diagnostic assistance unit 140 may be constructed by using a large amount of data and deep learning techniques, and then performing additional learning on the findings and diagnostic result suggestions limited to the field where the diagnostic support device 100 is used. In the case of findings where intermediate products are also input during additional learning, the filter unit 130 may add the intermediate products to the corrected feature information. Also, when image information is input during additional learning, the image information may be input to the diagnostic assistance unit 140. The diagnostic result suggestions are expected to play a role similar to a draft of the diagnostic result made by a doctor.

[0016] According to the diagnostic support device 100, by the filter unit 130 performing processing on the features obtained from the image information, appropriate information can be input to the diagnostic assistance unit 140. Therefore, a more accurate diagnostic result suggestion can be output. [Modification Example 1]

[0017] Fig. 3 shows a functional configuration example of the diagnostic support apparatus according to Modification Example 1, and Fig. 4 shows a processing flow example of the diagnostic support apparatus according to Modification Example 1. In the diagnostic support apparatus 101 according to Modification Example 1, the recording unit 191 records weights for each of a plurality of predetermined image features. The filter unit 131 obtains a weighted likelihood using the likelihood of the finding indicated by the feature information and the weight for the image feature included in the finding, and outputs the feature information selected based on the weighted likelihood as corrected feature information (S131). For example, the weighted likelihood may be obtained by multiplying the likelihood of the finding by the weight for the image feature included in the finding. However, it is not limited to simple multiplication, and the weighted likelihood may be obtained using an operation such that the weighted likelihood becomes larger for a larger weight if the likelihoods are the same. As described in Example 1, the "likelihood of the finding" means the probability (likelihood) that the image at the position indicated by the position-indicating information is an image feature (the above-mentioned word). When selecting based on the weighted likelihood, for example, a threshold common to all findings may be compared with the weighted likelihood, and the feature information with a larger weighted likelihood may be selected as the corrected feature information. In the case of this method, substantially the same filtering process as in Example 1, in which a threshold is set for each image feature, is performed. Alternatively, a predetermined number of combinations of findings and weighted likelihoods, starting from the ones with larger weighted likelihoods, may be used as the corrected feature information. The "predetermined number" may be one or a plurality such as three.

[0018] The input unit 110, the feature extraction unit 120, and the diagnosis assistance unit 140 are the same as those in Example 1. In the diagnostic support apparatus 101 as well, by the filter unit 131 performing processing on the image features obtained from the image information, appropriate findings can be input to the diagnosis assistance unit 140. Therefore, a more accurate diagnostic result proposal can be output. [Modification Example 2]

[0019] Figure 5 shows an example of the functional configuration of the diagnostic support device of Modification 2, and Figure 6 shows an example of the processing flow of the diagnostic support device of Modification 2. The feature extraction unit 122 of the diagnostic support device 102 of Modification 2 includes a model such as a convolutional neural network and outputs the findings on the image information and the intermediate products of the model such as the convolutional neural network as feature information (S122). Modification 2 differs from Example 1 and Modification 1 in that the feature extraction unit 122 does not output the likelihood of the findings.

[0020] The filter unit 132 outputs the intermediate product as corrected feature information if the findings correspond to predetermined findings (S132). The filter unit 132 may always output findings as corrected feature information, or it may output findings only when intermediate products are not output as corrected feature information. In Example 1 and Modification 1, filtering was performed based on likelihood, but in Modification 2, filtering is performed based on findings. Findings for which intermediate products are used as corrected feature information can be recorded in the recording unit 192.

[0021] The input unit 110 and the diagnostic support unit 140 are the same as in Example 1. In the diagnostic support device 102, the filter unit 132 processes the image features obtained from the image information, so that appropriate findings can be input to the diagnostic support unit 140. Therefore, a more accurate diagnostic result can be output.

[0022] [Processor, program, recording medium] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory.

[0023] In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein.

[0024] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0025] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 7, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0026] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0027] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0028] A computer executing such a program may, for example, first store the program recorded on a portable storage medium or a program transferred from a server computer in its own storage device. Then, when processing is to be executed, the computer reads the program stored on its own storage medium and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed by a so-called ASP (Application Service Provider) type service, where the processing function is realized only by execution instructions and result acquisition, without transferring the program from the server computer to this computer. Furthermore, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. In this form, the program includes information used for processing by an electronic computer that is equivalent to a program (data that is not a direct instruction to the computer but has the property of defining the computer's processing).

[0029] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware. [Explanation of Symbols]

[0030] 100, 101, 102 Diagnostic support device 110 Input section 120,122 Feature extraction section 130,131,132 Filter section 140 Diagnostic support unit 190,191,192 Recording unit

Claims

1. A recording unit that records threshold values ​​for each of several predetermined image features, An input unit that acquires pairs of text and image information related to the inspection, A feature extraction unit outputs findings on the aforementioned image information and feature information that indicates the likelihood of said findings. A filter unit that outputs the feature information as modified feature information when the likelihood of the finding indicated by the feature information exceeds the threshold for the image features included in the finding, A diagnostic support unit takes the aforementioned text information and the aforementioned modified feature information as input and outputs a proposed diagnostic result. A diagnostic support device equipped with the following features.

2. A recording unit that records the weights of each of the features of multiple predetermined images, An input unit that acquires pairs of text and image information related to the inspection, A feature extraction unit outputs findings on the aforementioned image information and feature information that indicates the likelihood of said findings. A filter unit that calculates a weighted likelihood using the likelihood of the finding indicated by the feature information and the weights for the image features included in the finding, and outputs the feature information selected based on the weighted likelihood as modified feature information. A diagnostic support unit takes the aforementioned text information and the aforementioned modified feature information as input and outputs a proposed diagnostic result. A diagnostic support device equipped with the following features.

3. A diagnostic support device according to claim 1 or 2, The feature extraction unit includes a model and also outputs intermediate products of the model as feature information. When the filter unit outputs predetermined findings as corrected feature information, it also outputs the intermediate product as corrected feature information. A diagnostic support device characterized by the following features.

4. An input unit that acquires pairs of text and image information related to the inspection, A feature extraction unit that includes a model and outputs the findings on the image information and the intermediate product of the model as feature information, If the aforementioned finding corresponds to a predetermined finding, a filter unit outputs the intermediate product as corrected feature information. A diagnostic support unit takes the aforementioned text information and the aforementioned modified feature information as input and outputs a proposed diagnostic result. A diagnostic support device equipped with the following features.

5. A diagnostic support device according to claim 4, The filter unit uses at least the above findings as modified feature information. A diagnostic support device characterized by the following features.

6. A diagnostic support device according to any one of claims 1, 2, or 4, The aforementioned findings are information indicating the characteristics of the image being examined and the location where those characteristics exist. A diagnostic support device characterized by the following features.

7. A diagnostic support device according to any one of claims 1, 2, or 4, The aforementioned diagnostic support unit also receives the aforementioned image information. A diagnostic support device characterized by the following features.

8. By defining threshold values ​​for each of several predetermined image features, An input step to obtain pairs of text and image information related to the examination, A feature extraction step that outputs findings on the aforementioned image information and feature information that indicates the likelihood of said findings, A filter step in which, when the likelihood of the finding indicated by the feature information exceeds the threshold for the image features included in the finding, the finding indicated by the feature information is output as modified feature information, A diagnostic support step that takes the aforementioned text information and the aforementioned modified feature information as input and outputs a proposed diagnostic result. A diagnostic support method for performing this task.

9. By defining weights for each of several predetermined image features, An input step to obtain pairs of text and image information related to the examination, A feature extraction step that outputs findings on the aforementioned image information and feature information that indicates the likelihood of said findings, A filter step which calculates a weighted likelihood using the likelihood of the finding indicated by the feature information and the weights for the image features included in the finding, and outputs the feature information selected based on the weighted likelihood as modified feature information, A diagnostic support step that takes the aforementioned text information and the aforementioned modified feature information as input and outputs a proposed diagnostic result. A diagnostic support method for performing this task.

10. An input step to obtain pairs of text and image information related to the examination, A feature extraction step that includes a model and outputs findings on the image information and intermediate products of the model as feature information, If the aforementioned finding corresponds to a predetermined finding, a filter step is performed to output the intermediate product as corrected feature information. A diagnostic support step that takes the aforementioned text information and the aforementioned modified feature information as input and outputs a proposed diagnostic result. A diagnostic support method for performing this task.

11. A diagnostic support program for causing a computer to function as a diagnostic support device according to any one of claims 1, 2, or 4.

Images