Report creation support device, report creation support method, and report creation support program

Abstract

Improving the workflow. [Solution] The report creation support device according to this embodiment includes an acquisition unit, a generation unit, and an input assistance unit. The acquisition unit acquires medical images. The generation unit uses multiple types of trained models that analyze images and output analysis results to generate the analysis results of the medical images and the confidence level of the analysis results for each trained model. The input assistance unit assists in inputting observation text based on the analysis results according to the confidence level.

Description

Technical Field

[0005] , , , , , , , ,

[0001] The embodiments disclosed in this specification and the drawings relate to a report creation support device, a report creation support method, and a report creation support program.

Background Art

[0002] <0​​​​​​​​​​​​​​​​​​​​​​​​​​​​​One of the problems that the embodiments disclosed herein and in the drawings aim to solve is to improve workflows. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0006] The report creation support device according to this embodiment includes an acquisition unit, a generation unit, and an input assistance unit. The acquisition unit acquires medical images. The generation unit uses multiple types of trained models that analyze images and output analysis results to generate analysis results of the medical images and a confidence level of the analysis results for each trained model. The input assistance unit assists in inputting observation text based on the analysis results according to the confidence level. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a block diagram showing a medical information system including a report creation support device. [Figure 2] Figure 2 is a block diagram of the report creation support device. [Figure 3] Figure 3 is a flowchart showing an example of the operation of the report creation support device. [Figure 4] Figure 4 shows a first example of a report template according to this embodiment. [Figure 5] Figure 5 shows a second example of a report template according to this embodiment. [Figure 6] Figure 6 shows an example of checking the input findings according to this embodiment. [Modes for carrying out the invention]

[0008] The report creation support device, report creation support method, and report creation support program according to this embodiment will be described below with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate.

[0009] The medical information system according to this embodiment will be described with reference to Figure 1. The medical information system according to this embodiment includes a report creation support device 1, an image server 2, an electronic medical record system 3, a medical information management application 4, and a report creation device 5, all of which are connected via a network.

[0010] In this embodiment, the report creation support device 1 is assumed to be separate from the image server 2, the electronic medical record system 3, the medical information management application 4, and the report creation device 5, but it may be included in at least one of the image server 2, the electronic medical record system 3, the medical information management application 4, and the report creation device 5.

[0011] The report creation support device 1 analyzes medical images using multiple types of pre-trained models, and adjusts the wording of the report template according to the image analysis results of the medical images and the corresponding confidence level, and presents it to users such as radiologists. Details of the report creation support device 1 will be described later with reference to Figure 2.

[0012] Image server 2 is, for example, a PACS (Picture Archiving and Communication System), a system that stores and manages medical image data. Image server 2 stores and manages medical image data converted according to the DICOM (Digital Imaging and Communication Medicine) standard, for example.

[0013] Electronic medical record system 3 is a system that stores and manages electronic medical record data, including patient information. Patient information includes, for example, patient ID, patient name, gender, age, medical history, lifestyle, and other information related to the electronic medical record, such as findings, disease name, vital signs, test results, clinical pathways, and treatment details. Clinical pathways represent standard treatment plans in chronological order.

[0014] Medical information management application 4 is an application that can integrate and manage patient information, particularly treatment details and examination information, on a timeline, and allows for the sharing of medical information among multiple doctors, or among users such as doctors, technicians, and nurses, who are all healthcare professionals.

[0015] The report creation device 5 is a device for creating a user-generated image interpretation report. In this embodiment, the report creation device 5 includes a display, and at least information transferred from the report creation support device 1 is displayed on the display, and the user generates an image interpretation report based on the transferred information.

[0016] The network is, for example, a hospital network. The network can be wired or wireless. Furthermore, the connection is not limited to the hospital network, as long as security is ensured. For example, it is acceptable to connect to public communication lines such as the internet via a VPN (Virtual Private Network).

[0017] Next, the details of the report creation support device 1 will be explained with reference to the block diagram in Figure 2. The report creation support device 1 shown in Figure 2 includes a processing circuit 10, a memory 11, an input interface 12, and a communication interface 13. The processing circuit 10, the memory 11, the input interface 12, and the communication interface 13 are connected to each other in a way that allows them to communicate with one another, for example, via a bus. The report creation support device 1 may also include a display.

[0018] The processing circuit 10 is a processor that functions as the center of the report creation support device 1. The processing circuit 10 includes an acquisition function 101, a generation function 102, an input assistance function 103, a display control function 104, and a feedback function 105.

[0019] The acquisition function 101 acquires medical images from, for example, the image server 2. The generation function 102 generates, for each learned model, an image analysis result of a medical image and a confidence level of the analysis result using a plurality of types of learned models that analyze an image and output an analysis result. The learned model is a machine learning model that performs image analysis processing of medical images. The learned model used in this embodiment is assumed to be a model that has already been trained to be capable of performing various image analysis processes. The learned model performs image processing that is useful for reading, such as registration processing, segmentation processing, extraction or highlighting of segmented regions, and measurement processing of the size or length of a target part (such as a region of interest). The confidence level is a value indicating how correct the image analysis result output by the learned model is. For example, in the case of segmentation processing, it is a ratio indicating the output accuracy of the learned model for the segmented region.

[0020] The input assistance function 103 performs input assistance that assists in inputting findings sentences based on the analysis result according to the confidence level. The input assistance function 103 generates, for example, a template of findings sentences with the text adjusted based on the confidence level of the analysis result. Further, the input assistance function 103 generates a confirmation message according to the confidence level for the findings sentences input by the user.

[0021] The display control function 104 displays, for example, the analysis result and the template of the findings sentences on, for example, the display of the report creation device 5. The feedback function 105 generates feedback information based on an input operation from the user for the template.

[0022] Memory 11 is a storage device such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), and integrated circuit storage device that stores various types of information. Memory 11 may also be a drive device that reads and writes various types of information to and from portable storage media such as CD-ROM drives, DVD drives, and flash memory. Note that memory 11 does not necessarily have to be implemented by a single storage device. For example, memory 11 may be implemented by multiple storage devices. Furthermore, memory 11 may be located in another device or another computer connected to the report creation support device 1 via a network.

[0023] Memory 11 stores the processing program and the like according to this embodiment. This program may, for example, be pre-stored in memory 11. Alternatively, for example, it may be stored and distributed on a non-transient storage medium, read from the non-transient storage medium, and installed in memory 11.

[0024] The input interface 12 receives various input operations from the user, converts the received input operations into electrical signals, and outputs them to the processing circuit 10. In this embodiment, the input interface 12 is connected to input devices such as a mouse, keyboard, trackball, switch, button, joystick, touchpad, and touch panel, where instructions are input by touching the operating surface. Alternatively, the input device connected to the input interface 12 may be an input device provided on another computer connected via a network or the like.

[0025] The communication interface 13 communicates data with the image server 2, the electronic medical record system 3, the medical information management application 4, the report creation device 5, and, although not shown, the hospital information system, the radiology department information system, etc. The communication interface 13 communicates data in accordance with, for example, a pre-configured known standard. Communication between the report creation support device 1 and the hospital information system, the electronic medical record system 3, the medical information management application 4, and the radiology department information system is, for example, conducted in accordance with HL7 (Health Level 7). Communication between the report creation support device 1 and the image server 2, the medical information management application 4, and the report creation device 5 is, for example, conducted in accordance with DICOM (Digital Imaging and Communications in Medicine).

[0026] Next, an example of the operation of the report creation support device 1 according to this embodiment will be explained with reference to the flowchart in Figure 3.

[0027] In step SA1, the processing circuit 10 acquires the medical image to be interpreted using the acquisition function 101.

[0028] In step SA2, the processing circuit 10 generates image analysis results for medical images using multiple types of pre-trained models via the generation function 102. The pre-trained models to be used may be pre-selected according to the types of image analysis that the pre-trained models can perform, automatically selected according to the type of examination, or manually selected by the user. Furthermore, the processing circuit 10 generates confidence scores for the image analysis results obtained from the pre-trained models via the generation function 102. Specifically, for example, if the pre-trained model performs segmentation processing, the output value (e.g., probability value) of the softmax function in multi-class classification may be used as the confidence score.

[0029] In step SA3, the input assistance function 103 enables the processing circuit 10 to generate candidate (template) findings statements based on the image analysis results and corresponding confidence levels. Specifically, for example, the input assistance function 103 enables the processing circuit 10 to generate templates using more definitive wording the higher the confidence level, and templates using more speculative wording the lower the confidence level. For example, the wording of the template may be created by classifying the confidence level into multiple stages, preparing a table that associates the wording of the observation statement with the confidence level for each stage, and extracting the wording corresponding to the confidence level from this table as a template. Alternatively, a trained model for template generation may be prepared, using the image analysis results and confidence levels as input data, and the observation statements entered by the user as ground truth data. By inputting the image analysis results and confidence levels generated in step SA2 into the trained model for template generation, the observation statements output by the trained model may be generated as templates. Furthermore, a large-scale language model such as GPT may be used. Thus, the generation of template text may be implemented using a rule-based method with tables, or it may be implemented using a machine learning model method with another pre-trained model, such as a large-scale language model like GPT. Furthermore, a combination of the rule-based method and the machine learning model method may also be used.

[0030] In step SA4, the processing circuit 10 displays the image analysis results and the template generated in step SA3 on, for example, the display of the report creation device 5, using the display control function 104. The processing circuit 10 may, using the display control function 104, prioritize displaying to the user, along with the template, image analysis results with higher confidence levels from among the multiple image analysis results. Alternatively, it may display only the image analysis result with the highest confidence level, or it may display a predetermined number of image analysis results in descending order of confidence level.

[0031] In step SA5, the processing circuit 10 acquires feedback information based on user input operations on the template via the feedback function 105. Specifically, the feedback information should be set so that the less or no user modifications are made to the template, the higher the output of the image analysis results of the trained model and the evaluation of the template.

[0032] In step SA6, the processing circuit 10 applies the feedback information to multiple trained models or templates via the feedback function 105. For example, if the user uses the template displayed in step SA4 as is, it can be said that the observation text is usable without modification, so the image analysis results and template of that trained model should be set and updated to be used preferentially. Alternatively, if there are significant modifications to the template displayed in step SA4, or if the wording of the template is not used, the priority of the image analysis results and template of that trained model may be lowered, and the image analysis results of other trained models and templates based on those analysis results may be given priority.

[0033] Furthermore, if the feedback information concerns modifications to the wording related to the formatting rather than the content of the observation statement represented by the template, it is not necessary to change the trained model and template. Alternatively, the processing circuit 10 may adjust the wording of the template for each user based on the feedback information using the feedback function 105. This allows the user's preferences to be reflected.

[0034] Next, we will explain the first example of a template that is adjusted according to the level of confidence, with reference to Figure 4.

[0035] Figure 4 shows the template and the corresponding confidence level of the image analysis result. Here, we assume the use of a trained model that performs image analysis for consolidation, i.e., shadows. As shown in Figure 4, when the confidence level is high, a template is generated that uses definitive wording such as "Patchy consolidation is observed in the left upper lobe of the lung." On the other hand, when the confidence level is low, a template is generated that uses speculative wording such as "Patchy consolidation is suspected in the left upper lobe of the lung" or "Patchy consolidation is possible in the left upper lobe of the lung." In this way, the verb part at the end of the template wording is changed according to the confidence level. Of course, this is not limited to Japanese; in English, when the confidence level is high, a definitive expression such as "Consolidation is observed..." can be used, and when the confidence level is low, an expression such as "Consolidation is suspected..." can be used.

[0036] In Figure 4, the classification of high or low confidence levels is based on the following criteria: if the confidence percentage is above a threshold, for example 90% or higher, it is judged as high confidence. On the other hand, if the confidence percentage is below a threshold, for example less than 60%, it is judged as low confidence. It is also possible to divide the confidence level into more stages, and adjust the wording of the template according to each stage.

[0037] Next, we will explain a second example of a template that is adjusted according to the level of confidence, with reference to Figure 5. Figure 5 shows an example of a template that combines the results of multiple image analyses. Specifically, it is a template that combines a first image analysis process related to segmentation and a second image analysis process related to estimating whether or not there is an infarct.

[0038] If the level of confidence is high, a template is generated that uses definitive wording, such as, "A faint high-attenuation area has appeared in the left basal ganglia, which is thought to be a hemorrhagic infarction." On the other hand, if the confidence level of the first image analysis is low, the result will be expressed as "left side" rather than "left basal ganglia," such as "A faint high-attenuation area has appeared on the left side, which is thought to be hemorrhagic infarction." Similarly, if the confidence level of the second image analysis is low, the wording regarding the infarction will be expressed in a speculative way, such as "A faint high-attenuation area has appeared in the left basal ganglia, suggesting the possibility of hemorrhagic infarction" or "A faint high-attenuation area is observed in the left basal ganglia." Furthermore, if the confidence level of both the first and second image analysis is low, the results of each image analysis will be expressed in a speculative way, such as "There is a suspicion of a high-attenuation area on the left side, and the possibility of hemorrhagic infarction cannot be ruled out."

[0039] Furthermore, the user may perform a confidence-based check on the findings entered by the user based on the image analysis results. An example of checking the entered findings is explained with reference to Figure 6.

[0040] For example, consider a scenario where a user reviews their findings based on low-confidence image analysis results, and the content of the findings statement, either without using a template or edited from a template, uses definitive language. In this case, the input assistance function 103 may cause the processing circuit 10 to display a confirmation message 60 such as "The analysis results have a low confidence level. Please check the wording of the findings." as shown in Figure 6.

[0041] As for the specific processing, for example, the input assistance function 103 causes the processing circuit 10 to acquire the observation text entered by the user at predetermined intervals and perform a comparison process between the acquired observation text and a template based on the confidence level of the image analysis results. This comparison process may involve comparing whether the strings of the observation text are an exact match or, for example, using a large-scale language model to determine whether the content of the observation text is definitive or speculative. If the comparison process determines that the acquired observation text and the template based on the confidence level of the image analysis results are different, the input assistance function 103 causes the processing circuit 10 to generate the confirmation message 60 described above. After that, the display control function 104 causes the processing circuit 10 to display the confirmation message 60. This allows users to be prompted to reconsider their judgments regarding the image processing results related to image analysis.

[0042] Furthermore, in addition to displaying confirmation message 60, the user may be presented with template wording based on the confidence level as a proposed revision to the observation statement.

[0043] According to the embodiment described above, the generation function uses multiple types of trained models to analyze images and output analysis results, generating image analysis results and confidence levels of the analysis results for each trained model of a medical image. The input assistance function provides input assistance to help input a report based on the analysis results, according to the confidence level. Specifically, it generates a report template with wording adjusted based on the confidence level of the analysis results. Alternatively, it generates a confirmation message for the report entered by the user, according to the confidence level. This provides users, such as radiologists, with templates based on image analysis results when creating report texts. Users can then use these templates to create report texts, reducing the time spent on both image interpretation and report creation. As a result, the workflow can be improved.

[0044] In the above description, the term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is a CPU, for example, it performs its functions by reading and executing a program stored in a memory circuit. On the other hand, when the processor is an ASIC, for example, instead of the program being stored in a memory circuit, the function is directly incorporated as a logic circuit within the processor's circuit. In this embodiment, each processor is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and perform its functions. Furthermore, multiple components shown in the figure may be integrated into a single processor to perform its functions.

[0045] In addition, each function according to the embodiment can also be realized by installing a program that performs the processing on a computer such as a workstation and loading it into memory. In this case, the program that can cause the computer to execute the method can also be stored and distributed on a storage medium such as a magnetic disk (hard disk, etc.), optical disk (CD-ROM, DVD, etc.), or semiconductor memory.

[0046] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0047] 1. Report creation support device 2 Image Server 3. Electronic medical record system 4. Medical Information Management Application 5. Report generation device 10 Processing Circuit 11 memory 12 Input Interfaces 13 Communication Interface 60 Confirmation Message 101 Acquisition function 102 Generation function 103 Input assistance function 104 Display control function 105 Feedback function

Claims

1. The acquisition unit acquires medical images, A generation unit that uses multiple types of pre-trained models to analyze images and output analysis results, and generates the analysis results of the medical image and the confidence level of the analysis results for each pre-trained model, An input assistance unit assists in inputting observation statements based on the analysis results, according to the aforementioned confidence level, A report creation support device equipped with the following features.

2. The report creation support device according to claim 1, wherein the input assistance unit generates a template in which the wording is adjusted according to the degree of confidence for a template of observations based on at least one of the analysis results.

3. The report creation support device according to claim 2, wherein the input assistance unit generates the template using definitive wording that increases with the degree of confidence.

4. The report creation support device according to claim 2, wherein the input assistance unit generates the template using speculative wording, the lower the degree of confidence.

5. The system further comprises a display control unit for displaying the analysis results, The report creation support device according to claim 1, wherein the input assistance unit generates a confirmation message for the observation text entered by the user according to the confidence level corresponding to the analysis result.

6. The report creation support device according to claim 1, further comprising a display control unit that prioritizes displaying the corresponding analysis result and a template with wording adjusted according to the confidence level, the higher the confidence level.

7. The report creation support device according to claim 2, further comprising a feedback unit that generates feedback information relating to at least one of the trained model that outputs the analysis results and the template, based on user input operations on the template.

8. The report creation support device according to claim 7, wherein the feedback unit sets the feedback information such that the evaluation of at least one of the trained model that output the analysis results and the template becomes higher the less or no modifications are made by the user to the template.

9. The acquisition method involves acquiring medical images, The generation means uses multiple types of trained models that analyze images and output analysis results to generate the analysis results of the medical image and the confidence level of the analysis results for each trained model. A report creation support method comprising an input assistance means that assists in inputting observations based on the analysis results according to the confidence level.

10. On the computer, Medical image acquisition function, A generation function that uses multiple types of pre-trained models to analyze images and output analysis results, and generates the analysis results of the medical images and the confidence level of the analysis results for each pre-trained model, A report creation support program that provides an input assistance function to assist in inputting observations based on the analysis results, according to the aforementioned confidence level.

Images