Document creation support device, method, and program

Abstract

To provide a document creation assistance device and method and a program, in which sentences regarding a structure of interest that is included in an image are generated from the image with high accuracy.SOLUTION: A document creation assistance device includes at least one processor. The processor is constituted so as to analyze an image and thereby derive property information that represents the property of a structure of interest included in the image, generate a sentence regarding the image on the basis of the property information, analyze the sentence so as to identify the term that represents the property regarding the structure of interest that is included in the sentence, and verify the property information against the term.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a document creation support device, method, and program that support the creation of documents such as medical documents. [Background technology]

[0002] In recent years, advances in medical equipment such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) devices have made it possible to perform diagnostic imaging using higher quality, higher resolution medical images. In particular, diagnostic imaging using CT and MRI images can pinpoint the area of ​​a lesion with high accuracy, and appropriate treatment can now be performed based on the results of this identification.

[0003] In addition, medical images are analyzed using CAD (Computer-Aided Diagnosis) that uses a learning model that has been machine-learned using deep learning, etc., to determine the shape, concentration, position, size, and other characteristics of structures of interest, such as abnormal shadow candidates, contained in the medical images, and these are obtained as analysis results. The analysis results obtained by CAD are associated with examination information, such as the patient's name, sex, age, and the modality that obtained the medical image, and are stored in a database. The medical images and the analysis results are transmitted to the terminal of the radiologist who interprets the medical images. The radiologist interprets the medical images on his or her own terminal by referring to the transmitted medical images and analysis results, and creates an interpretation report.

[0004] On the other hand, with the improvement of the performance of the above-mentioned CT device and MRI device, the number of medical images to be interpreted is also increasing. However, since the number of doctors who interpret images is not keeping up with the number of medical images, it is desired to reduce the burden of doctors who interpret images. For this reason, various methods have been proposed to support the creation of medical documents such as image interpretation reports. For example, Patent Document 1 proposes a method of generating text to be written in an image interpretation report based on information (hereinafter referred to as property information) that represents the properties of a structure of interest contained in keywords input by an image interpretation doctor and analysis results of medical images (see Patent Document 1). In the method described in Patent Document 1, medical text (hereinafter referred to as medical text) is created using a learning model that has been machine-learned, such as a recurrent neural network that has been trained to generate text from characters that represent the input property information. By automatically generating medical text as in the method described in Patent Document 1, the burden on doctors who interpret images when creating medical documents such as image interpretation reports can be reduced. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-153250 A Summary of the Invention [Problem to be solved by the invention]

[0006] On the other hand, it is conceivable to generate a learning model that generates medical text using only the property information acquired by analyzing medical images. However, depending on the contents of the teacher data used to train the learning model, or depending on the learning limit of the learning model, all property information acquired from the medical image may not be included in the generated medical text. In addition, property information other than the property information acquired from the medical image may be included in the generated medical text. If such medical text is used as a radiology report, the accuracy of the radiology report will decrease.

[0007] The present disclosure has been made in consideration of the above circumstances, and aims to generate text regarding a structure of interest contained in an image with high accuracy from the image, such as when generating medical text from a medical image. [Means for solving the problem]

[0008] A document creation support device according to the present disclosure includes at least one processor, The processor By analyzing the image, property information that represents the properties of a structure of interest included in the image is derived;

[0009] Generate text about the image based on the property information; By analyzing the text, terms that express properties related to the structure of interest contained in the text are identified; The system is configured to match the property information with terms.

[0010] In addition, in the document creation support device according to the present disclosure, the processor may be further configured to display the text on the display.

[0011] In this case, the processor may be further configured to display the result of the matching on a display.

[0012] In addition, in the document creation support device according to the present disclosure, the processor may be further configured to display the result of the comparison by highlighting the difference between the term and the property information in the sentence when the term and the property information differ.

[0013] In addition, in the document creation support device according to the present disclosure, the processor may be further configured to regenerate the sentence when the term and the property information differ in the sentence.

[0014] In addition, in the document creation support device according to the present disclosure, the processor may be further configured to accept corrections to the text.

[0015] In addition, in the document creation support device according to the present disclosure, the processor generates a plurality of sentences related to the image based on the property information, Analyzing each of the plurality of sentences to identify terms that are included in each of the plurality of sentences and that describe properties related to the structure of interest; Matching the property information with the term for each of the plurality of sentences; The apparatus may be configured to select at least one sentence from the plurality of sentences based on a result of the matching.

[0016] In addition, in the document creation support device according to the present disclosure, the image may be a medical image, and the text may be medical text relating to a structure of interest contained in the medical image.

[0017] The document creation support method according to the present disclosure includes: analyzing an image to derive property information that represents properties of a structure of interest included in the image; Generate text about the image based on the property information; By analyzing the text, terms that express properties related to the structure of interest contained in the text are identified; Match property information with terms.

[0018] The document creation support method according to the present disclosure may be provided as a program for causing a computer to execute the method. Effect of the Invention

[0019] According to the present disclosure, text regarding structures of interest contained in an image can be generated with high accuracy from the image. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical information system to which a document creation support device according to an embodiment of the present disclosure is applied; [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of a document creation support device according to an embodiment of the present invention; [Diagram 3]FIG. 1 is a diagram showing an example of training data for learning a first learning model. [Figure 4] A schematic diagram of a recurrent neural network. [Diagram 5] A diagram showing an example of training data for learning the third learning model. [Figure 6] A diagram showing an example of training data for learning the fourth learning model. [Figure 7] A diagram showing an example of a display screen for medical documents and matching results. [Figure 8] A diagram showing an example of a display screen for medical documents and matching results. [Figure 9] A diagram showing an example of a display screen for medical documents and matching results. [Figure 10] A flowchart showing the processing performed in this embodiment. [Figure 11] A flowchart showing the process performed in another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. FIG. 1 is a diagram showing a schematic configuration of a medical information system to which a document creation support device according to an embodiment of the present disclosure is applied. The medical information system 1 shown in FIG. 1 is a system for performing imaging of a subject's examination target site, storing the medical images acquired by imaging, interpretation of the medical images by a radiologist and creation of an interpretation report, and browsing of the interpretation report and detailed observation of the medical image to be interpreted by a requesting doctor of the medical department, based on an examination order from a doctor of a medical department using a known ordering system. As shown in FIG. 1, the medical information system 1 is configured by connecting a plurality of modalities (imaging devices) 2, a plurality of interpretation workstations (WS) 3 as interpretation terminals, a medical department workstation (WS) 4, an image server 5, an image database 6, an interpretation report server 7, and an interpretation report database 8 in a state in which they can communicate with each other via a wired or wireless network 10.

[0022] Each device is a computer installed with an application program for functioning as a component of the medical information system 1. The application program is stored in a state accessible from the outside in a storage device of a server computer connected to the network 10 or in a network storage, and is downloaded to the computer upon request and installed. Alternatively, the application program is recorded on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) and distributed, and is installed on the computer from the recording medium.

[0023] The modality 2 is a device that captures an image of a diagnostic target part of a subject, thereby generating a medical image representing the diagnostic target part. Specifically, the modality 2 is a plain X-ray device, a CT device, an MRI device, a PET (Positron Emission Tomography) device, etc. The medical image generated by the modality 2 is transmitted to the image server 5 and stored therein.

[0024] The image interpretation WS 3 includes a document creation support device according to this embodiment. The configuration of the image interpretation WS 3 will be described later.

[0025] The department WS4 is a computer used by doctors in the department to observe images in detail, view image interpretation reports, create electronic medical records, etc., and is composed of a processing device, a display, and input devices such as a keyboard and a mouse. The department WS4 executes software programs for each process, such as creating patient records (electronic medical records), requesting the image server 5 to view images, displaying images received from the image server 5, automatically detecting or highlighting parts that appear to be lesions in images, requesting the image interpretation report server 7 to view image interpretation reports, and displaying image interpretation reports received from the image interpretation report server 7.

[0026] The image server 5 is a general-purpose computer installed with a software program that provides the functions of a database management system (DBMS). The image server 5 also includes a storage in which the image database 6 is configured. This storage may be a hard disk device connected to the image server 5 via a data bus, or a disk device connected to a NAS (Network Attached Storage) or SAN (Storage Area Network) that is connected to the network 10. When the image server 5 receives a request to register a medical image from the modality 2, it prepares the medical image in a database format and registers it in the image database 6.

[0027] Image data and additional information of medical images acquired by the modality 2 are registered in the image database 6. The additional information includes, for example, an image ID (identification) for identifying each medical image, a patient ID for identifying the subject, an examination ID for identifying the examination, a unique ID (UID: unique identification) assigned to each medical image, the examination date and time when the medical image was generated, the type of modality used in the examination to acquire the medical image, patient information such as the patient's name, age, and sex, the examination site (imaging site), imaging information (imaging protocol, imaging sequence, imaging technique, imaging conditions, use of contrast agent, etc.), and a series number or collection number when multiple medical images are acquired in one examination.

[0028] Furthermore, when the image server 5 receives a viewing request from the image interpretation WS 3 via the network 10, it searches for medical images registered in the image database 6, and transmits the searched medical images to the image interpretation WS 3 that has made the request.

[0029] A software program that provides a general-purpose computer with the functions of a database management system is installed in the image interpretation report server 7. When the image interpretation report server 7 receives a registration request for an image interpretation report from the image interpretation WS 3, it converts the image interpretation report into a database format and registers it in the image interpretation report database 8. Also, when it receives a search request for an image interpretation report, it searches the image interpretation report database 8 for the image interpretation report.

[0030] The image reading report database 8 stores image reading reports that include information such as an image ID for identifying the medical image being read, an image reading physician ID for identifying the image diagnostician who performed the image reading, the name of the lesion, location information of the lesion, findings, and the certainty of the findings.

[0031] In this embodiment, the medical image is a three-dimensional CT image consisting of multiple cross-sectional images of the lungs as a diagnostic target, and the CT image is interpreted to create a medical document, such as an interpretation report on abnormal shadows in the lungs. The medical image is not limited to a CT image, and any medical image, such as an MRI image or a simple two-dimensional image obtained by a simple X-ray device, can be used.

[0032] The network 10 is a wired or wireless network that connects various devices in the hospital. If the image interpretation WS3 is installed in other hospitals or clinics, the network 10 may be configured by connecting the local area networks of the hospitals via the Internet or a dedicated line.

[0033] The image interpretation WS3 according to the present embodiment will be described in detail below. The image interpretation WS3 is a computer used by a doctor who interprets medical images to interpret medical images and create image interpretation reports, and is composed of a processing device, a display, and input devices such as a keyboard and a mouse. In the image interpretation WS3, various processes such as a request to view a medical image to the image server 5, various image processing for a medical image received from the image server 5, display of a medical image, analysis processing for a medical image, highlighting of a medical image based on an analysis result, creation of an image interpretation report based on an analysis result, support for creating an image interpretation report, a request to register and view an image interpretation report to the image interpretation report server 7, and display of an image interpretation report received from the image interpretation report server 7 are performed by executing software programs for each process. Among these processes, processes other than those performed by the document creation support device of the present embodiment are performed by well-known software programs, and detailed descriptions are omitted here. In addition, processing other than that performed by the document creation support device of this embodiment may not be performed in the interpretation WS3, but a separate computer that performs that processing may be connected to the network 10, and the requested processing may be performed on that computer in response to a processing request from the interpretation WS3.

[0034] The image reading WS3 includes a document creation support device according to the present embodiment. Therefore, the image reading WS3 has a document creation support program according to the present embodiment installed therein. The document creation support program is stored in a storage device of a server computer connected to a network or in a network storage in a state accessible from the outside, and is downloaded and installed in the image reading WS3 upon request. Alternatively, the document creation support program is recorded on a recording medium such as a DVD or CD-ROM and distributed, and is installed in the image reading WS3 from the recording medium.

[0035] Fig. 2 is a diagram showing a schematic configuration of a document creation support device according to this embodiment, which is realized by installing a document creation support program in an interpretation WS 3. As shown in Fig. 2, the document creation support device 20 includes a CPU (Central Processing Unit) 11, a memory 12, a storage 13, and a communication I / F (interface) 14 as a standard computer configuration. In addition, a display 15 such as a liquid crystal display, and an input device 16 such as a keyboard and a mouse are connected to the document creation support device 20. The CPU 11 corresponds to a processor.

[0036] The storage 13 is a storage device such as a hard disk or an SSD (Solid State Drive). The storage 13 stores various information including information necessary for processing medical images and the document creation support device 20, which is acquired from the image server 5 via the network 10.

[0037] The communication I / F 14 is a network interface that controls the transmission of various information between an external device and the document creation support device 20 via the network 10 .

[0038] A document creation support program is also stored in the memory 12. The document creation support program specifies, as processes to be executed by the CPU 11, an image acquisition process for acquiring a medical image, an image analysis process for analyzing the medical image to derive property information representing the property of a structure of interest contained in the medical image, a text generation process for generating medical text related to the medical image based on the property information, a term identification process for analyzing the medical text to identify terms representing the property of the structure of interest contained in the medical text, a comparison process for comparing the property information with terms, and a display control process for displaying the medical text and the comparison result on the display 15.

[0039] When the CPU 11 executes these processes in accordance with the document creation support program, the computer functions as an image acquisition unit 21, an image analysis unit 22, a sentence generation unit 23, a term identification unit 24, a matching unit 25, and a display control unit .

[0040] The image acquisition unit 21 is composed of an interface connected to the network 10, and acquires medical images for creating an image interpretation report from the image server 5 in response to an instruction from the input device 16 by an operator, that is, an image interpretation doctor.

[0041] The image analysis unit 22 derives property information that indicates the property of a structure of interest, such as an abnormal shadow candidate, included in the medical image by analyzing the medical image. For this purpose, the image analysis unit 22 has a first learning model 22A that has been machine-learned to determine whether or not each pixel (voxel) in the medical image represents an abnormal shadow candidate, and if it is an abnormal shadow candidate, to determine its property. In this embodiment, the first learning model 22A is a convolutional neural network (CNN) that has been deep-learned using teacher data to determine whether or not each pixel (voxel) in the medical image represents an abnormal shadow candidate, and if it is an abnormal shadow candidate, to determine its property.

[0042] Fig. 3 is a diagram showing an example of training data for training the first learning model. As shown in Fig. 3, training data 30 includes a medical image 32 including an abnormal shadow 31 and characteristic information 33 about the abnormal shadow. In this embodiment, the abnormal shadow 31 is a pulmonary nodule, and the characteristic information 33 indicates a plurality of characteristics about the pulmonary nodule. For example, the characteristic information 33 may include the location of the abnormal shadow, the size of the abnormal shadow, the shape of the boundary (clear and irregular), the type of absorption value (solid and ground-glass), the presence or absence of spicules, whether it is a mass or a nodule, the presence or absence of pleural contact, the presence or absence of pleural indentation, the presence or absence of pleural infiltration, the presence or absence of cavities, and the presence or absence of calcification. As for the abnormal shadow 31 included in the training data 30 shown in FIG. 3, the characteristic information 33 indicates that the location of the abnormal shadow is under the left lung pleura, the size of the abnormal shadow is 4.2 cm in diameter, the shape of the boundary is irregular, the absorption value is solid, there is spicula, there is mass and pleural contact, there is pleural indentation, there is no pleural infiltration, there is no cavity, and there is no calcification. In FIG. 3, "present" is indicated by +, and "absent" is indicated by -. The first learning model 22A is constructed by training a neural network using a large number of training data as shown in FIG. 3. For example, by using the training data 30 shown in FIG. 3, when a medical image 32 shown in FIG. 3 is input, the first learning model 22A is trained to identify the abnormal shadow 31 included in the medical image 32 and output the characteristic information 33 shown in FIG. 3 for the abnormal shadow 31.

[0043] As the first learning model 22A, in addition to a convolutional neural network, any learning model such as a support vector machine (SVM) can be used.

[0044] A learning model for detecting abnormal shadow candidates from medical images and a learning model for detecting property information of the abnormal shadow candidates may be constructed separately.

[0045] The text generation unit 23 generates medical text using the characteristic information derived by the image analysis unit 22. The text generation unit 23 is composed of a second learning model 23A that has been trained to generate text from input information. As the second learning model 23A, for example, a recurrent neural network can be used. FIG. 4 is a diagram showing a schematic configuration of a recurrent neural network. As shown in FIG. 4, the recurrent neural network 40 is composed of an encoder 41 and a decoder 42. The characteristic information derived by the image analysis unit 22 is input to the encoder 41. For example, the characteristic information of "left lung subpleural", "4.2 cm", "spicula+" and "tumor" is input to the encoder 41. The decoder 42 has been trained to convert character information into text, and generates text from the input characteristic information. Specifically, from the above-mentioned characteristic information of "left lung subpleural", "4.2cm", "spicule+" and "tumor", a medical sentence of "4.2cm diameter tumor with spicules is observed in the left lung subpleural" is generated. In Figure 4, "EOS" indicates the end of the sentence (End Of Sentence).

[0046] In this way, in order to output medical text in response to input of characteristic information, the recurrent neural network 40 is constructed by training the encoder 41 and the decoder 42 using a large amount of training data consisting of combinations of characteristic information and medical text.

[0047] It should be noted that it is possible to specify terms that should not be used in a sentence and terms that should be used in a sentence as parameters for generating sentences in the recurrent neural network 40. These parameters are determined based on the results of matching by the matching unit 25, which will be described later.

[0048] The term identification unit 24 identifies terms expressing properties contained in the medical text generated by the text generation unit 23. For this purpose, the term identification unit 24 has a third learning model 24A that has been subjected to machine learning to identify terms expressing properties contained in the text. In this embodiment, the third learning model 24A is made up of a convolutional neural network that has been subjected to deep learning using teacher data so as to determine, when a text is input, terms expressing properties contained in the input text.

[0049] FIG. 5 is a diagram showing an example of training data for training the third learning model. As shown in FIG. 5, the training data 50 includes a medical text 51 and a term 52 expressing a characteristic contained in the medical text 51. The medical text 51 shown in FIG. 5 is "A solid mass with a clear boundary is observed in the left lower lobe S6 of the lung." The terms expressing the characteristic 52 are "left lower lobe S6 of the lung," "clear boundary," "solid," and "mass" contained in the medical text 51. The third learning model 24A is constructed by training a neural network using a large amount of training data as shown in FIG. 5. For example, by using the training data 50 shown in FIG. 5, the third learning model 24A is trained to output the term 52 shown in FIG. 5 when the medical text 51 shown in FIG. 5 is input.

[0050] As the third learning model 24A, in addition to the convolutional neural network, any learning model such as a support vector machine and a recurrent neural network can be used.

[0051] The collation unit 25 collates the characteristic information derived by the image analysis unit 22 with the terms expressing the characteristics included in the medical text identified by the term identification unit 24. For this purpose, the collation unit 25 has a fourth learning model 25A that has been machine-learned to distinguish the difference between the terms expressing the characteristics and the characteristic information when the characteristic information and the terms expressing the characteristics are input, and to distinguish the characteristic information that is not included in the terms expressing the characteristics, and the terms expressing the characteristics that are not included in the characteristic information. In this embodiment, the fourth learning model 25A is made up of a convolutional neural network that has been deep-learned using teacher data, so as to distinguish the difference between the terms expressing the characteristics and the characteristic information, and to distinguish the characteristic information that is not included in the terms expressing the characteristics, and the terms expressing the characteristics that are not included in the characteristic information.

[0052] FIG. 6 is a diagram showing an example of teacher data for learning the fourth learning model. Two types of teacher data 60 and 65 are shown in FIG. 6. As shown in FIG. 6, teacher data 60 includes attribute information 61, terms 62 representing attributes, and excess / deficiency information 63 representing excess / deficiency of attribute information. Teacher data 65 includes attribute information 66, terms 67 representing attributes, and excess / deficiency information 68 representing excess / deficiency of attribute information. Regarding teacher data 60 shown in FIG. 6, attribute information 61 is "left lung lower lobe S6", "border: clear", "absorption value: solid type", "spicula+", and "tumor". Terms 62 are "left lung lower lobe S6", "border clear", "solid type", and "tumor". Excess / deficiency information 63 is "insufficient spicules".

[0053] With regard to the teacher data 65 shown in Fig. 6, the characteristic information 66 is, like the characteristic information 61, "left lung lower lobe S6", "border: clear", "absorption value: solid type", "spicules +", and "tumor". The terms 67 are "left lung lower lobe S6", "border clear", "solid type", "spicules found", "calcification found", and "tumor". The excess / deficiency information 68 is "excessive calcification".

[0054] The fourth learning model is constructed by training a neural network using a large amount of training data as shown in Fig. 6. For example, by using training data 60 shown in Fig. 6, the fourth learning model 25A is trained to output surplus / deficiency information 63 when property information 61 and property term 62 shown in Fig. 6 are input. In addition, based on the surplus / deficiency information 63, the fourth learning model 25A is trained to generate a parameter indicating that the term "spicula" is lacking as a parameter based on the matching result.

[0055] Moreover, by using the teacher data 65 shown in Fig. 6, the fourth learning model 25A is trained to output excess / deficiency information 68 when property information 66 and property term 67 shown in Fig. 6 are input. Also, based on the excess / deficiency information 68, the fourth learning model 25A is trained to generate a parameter indicating that the term "calcification" is excessive as a parameter based on the matching result.

[0056] As the fourth learning model 25A, in addition to the convolutional neural network, any machine learning model such as a support vector machine and a recurrent neural network can be used.

[0057] The display control unit 26 displays the medical text generated by the text generation unit 23 and the matching result by the matching unit 25 on the display 15. FIG. 7 is a diagram showing an example of a display screen for the medical text and the matching result. As shown in FIG. 7, a display screen 70 includes an image display area 71 and a text display area 72. The image display area 71 displays a slice image SL1 in which the abnormal shadow candidate detected by the image analysis unit 22 is most easily identified. The slice image SL1 includes an abnormal shadow candidate 73, and the abnormal shadow candidate 73 is surrounded by a rectangular area 74.

[0058] In the text display area 72, medical text 75 generated by the text generation unit 23 and characteristic information 76 derived by the image analysis unit 22 are displayed. Note that the characteristic information 76 does not have to be displayed. The medical text 75 is "A solid mass with a clear boundary is observed in the left lower lobe S6 of the lung. Spicules and calcifications are observed." The characteristic information 76 is "Left lower lobe S6 of the lung," "Boundary: clear," "Absorption value: solid," "Spicula: +," and "Mass." In addition, as a result of the matching, since the "calcification" contained in the medical text 75 is excessive, a solid line frame 77 indicating that it is excessive is added to the term "calcification" contained in the medical text 75, and "calcification" is highlighted, thereby displaying the matching result.

[0059] Below the text display area 72, an automatic correction button 78A, a manual correction button 78B, and a confirmation button 78C are displayed.

[0060] The image interpreting doctor interprets the slice image SL1 displayed in the image display area 71 and judges the appropriateness of the medical text 75 displayed in the text display area 72. In addition, the image interpreting doctor can recognize that the term "calcification" included in the medical text 75 is excessive because of the solid line frame 77 attached to "calcification."

[0061] On the other hand, if the image interpretation doctor wishes to re-create the medical text 75, he / she selects the automatic correction button 78A using the input device 16. This causes the text generation unit 23 to re-create the medical text using the characteristic information derived by the image analysis unit 22. At this time, a parameter based on the collation result, that is, a parameter for not using the term "calcification", is input to the second learning model 23A of the text generation unit 23. This allows the text generation unit 23 to generate medical text without using the term "calcification", and as a result, it becomes possible to generate medical text such as "A solid mass with a clear boundary is observed in the left lower lobe S6. Spicules are observed." In this case, the collation result shown in FIG. 9, which will be described later, is displayed.

[0062] On the other hand, by selecting the manual correction button 78B, the image interpreting doctor can manually correct the medical text 75 displayed in the text display area 72 by inputting from the input device 16. Also, by selecting the confirm button 78C, the medical text 75 displayed in the text display area 72 can be confirmed as it is. As a result, the medical text 75 is transcribed into the image interpretation report, and the image interpretation report into which the medical text 75 has been transcribed is transmitted to the image interpretation report server 7 together with the slice image SL1 and stored therein.

[0063] Fig. 8 is a diagram showing another example of the display screen of the medical text and the matching result. In the display screen shown in Fig. 8, the same components as those in Fig. 7 are given the same reference numerals, and detailed explanations are omitted. In the text display area 72 of the display screen 70A shown in Fig. 8, medical text 75A is displayed, which reads, "A solid tumor mass with a clear boundary has been found in the left lower lobe S6."

[0064] Here, the characteristic information 76 is "left lung lower lobe S6", "boundary: clear", "absorption value: solid", "spicula: +", and "tumor". In addition, since the medical text 75A lacks the term "spicula", the term "spicula" is displayed in the text display area 72 and is highlighted with a dashed frame 79 indicating the lack of the term, thereby displaying the matching result.

[0065] Regarding the display screen 70A shown in FIG. 8, the radiologist interprets the abnormal shadow candidate 73 in the slice image SL1 displayed in the image display area 71 and judges the suitability of the medical text 75A displayed in the text display area 72. The radiologist can recognize that the term "spicula" included in the medical text 75A is lacking from the dashed frame 79 attached to "spicula". Instead of or in addition to displaying the term "spicula" in the text display area 72 and attaching the dashed frame 79 indicating the lack, a mark such as an arrow or a frame indicating the lack of the term "spicula" displayed in the characteristic information 76 may be attached. FIG. 9 shows a state in which an arrow 81 indicating the lack of the term "spicula" in the medical text 75A is further attached as a mark to "spicula" displayed in the characteristic information 76.

[0066] If the radiologist wishes to re-create the medical text 75A, he / she selects the automatic correction button 78A using the input device 16. This causes the text generation unit 23 to re-create the medical text using the property information derived by the image analysis unit 22. At this time, a parameter based on the collation result, i.e., a parameter for using the term "spicula", is input to the learning model of the text generation unit 23. This causes the text generation unit 23 to generate medical text using the term "spicula", and as a result, it becomes possible to generate medical text such as "A solid mass with a clear boundary is found in the left lower lobe S6. Spicules are found." In this case, the collation result shown in FIG. 9, which will be described later, is displayed. In this case, the medical text is generated by adding the text "Spicula is found" to the medical text already created, i.e., "A solid mass with a clear boundary is found in the left lower lobe S6." However, the entire medical text 75 may be generated again.

[0067] When the medical text displayed in the text display area 72 displays the proper information on the characteristics, for example, as in the display screen 70B shown in Fig. 9, an OK mark 80 is displayed in the text display area 72. The OK mark 80 enables the radiologist to recognize that the medical text 75B displayed in the text display area 72, "A solid tumor with a clear boundary is observed in the left lower lobe S6. Spicules are observed," uses the proper information on the characteristics derived from the medical image.

[0068] Next, the process performed in this embodiment will be described. FIG. 10 is a flowchart showing the process performed in this embodiment. It is assumed that the medical image to be interpreted is acquired from the image server 5 by the image acquisition unit 21 and stored in the storage 13. The process is started when an instruction to create an interpretation report is given by the interpreting doctor, and the image analysis unit 22 derives attribute information representing the attributes of a structure of interest such as an abnormal shadow candidate contained in the medical image by analyzing the medical image (step ST1). Next, the text generation unit 23 generates medical text related to the medical image based on the attribute information (step ST2). Next, the term identification unit 24 analyzes the medical text generated by the text generation unit 23 to identify terms representing the attributes of the structure of interest contained in the medical text (step ST3). Furthermore, the collation unit 25 collates the attribute information derived by the image analysis unit 22 with the terms identified by the term identification unit 24 (step ST4). Then, the display control unit 26 displays the medical text generated by the text generation unit 23 and the collation result by the collation unit 25 on the display 15 (step ST5).

[0069] Next, the display control unit 26 judges whether or not the automatic correction button 78A displayed on the display screen of the matching result has been selected (step ST6). If step ST6 is positive, the process returns to step ST2, and the processes of steps ST2 to ST5 are repeated. As a result, the text generation unit 23 regenerates the medical text according to parameters based on the matching result. The term identification unit 24 identifies terms in the regenerated medical text. The matching unit 25 matches the characteristic information with the re-identified terms. The display control unit 26 displays a display screen including the re-matching result on the display 15.

[0070] If step ST6 is negative, the display control unit 26 judges whether the manual correction button 78B displayed on the display screen of the matching result is selected (step ST7). If step ST7 is positive, the display control unit 26 accepts corrections made using the input device 16 to the medical text displayed in the text display area 72 (step ST8).

[0071] If step ST7 is negative, and following step ST8, the display control unit 26 determines whether the confirm button 78C is selected (step ST9). If step ST9 is negative, the process returns to step ST6. If step ST9 is positive, the display control unit 26 transcribes the medical text into an interpretation report, and transmits the interpretation report into which the medical text has been transcribed together with the slice image SL1 to the interpretation report server 7 (interpretation report transmission: step ST10), and ends the process.

[0072] In this manner, in this embodiment, by analyzing a medical image, attribute information expressing the attributes of a structure of interest, such as an abnormal shadow candidate, contained in the medical image is derived, and medical text related to the medical image is generated based on the attribute information. In addition, by analyzing the medical text, terms expressing the attributes of the structure of interest contained in the medical text are identified, and the attribute information and the terms are collated. Therefore, by referring to the collation result, it is possible to easily check whether the medical text has been generated using all the attribute information or whether excessive attribute information is not included in the text. As a result, by referring to the collation result, the medical text can be corrected or the medical text can be generated again to improve the accuracy of the generated medical text. Therefore, according to this embodiment, medical text related to the structure of interest contained in the medical image can be generated with high accuracy from the medical image.

[0073] Next, another embodiment of the present disclosure will be described. The configuration of the document creation support device according to the other embodiment is the same as the configuration of the document creation support device 20 shown in FIG. 2, and only the processing performed is different, so detailed description of the device will be omitted here. The document creation support device according to the other embodiment differs from the above embodiment in that the text generation unit 23 generates multiple medical texts based on property information. FIG. 11 is a flowchart showing the processing performed in the other embodiment. In the other embodiment, the processing up to step ST5 shown in FIG. 10 is different from the above embodiment. Therefore, in FIG. 11, only the differences from the flowchart shown in FIG. 10 will be described.

[0074] The process is started when an instruction to create an image interpretation report is given by the image interpretation doctor, and the image analysis unit 22 derives attribute information that represents the attributes of structures of interest, such as abnormal shadow candidates, contained in the medical image by analyzing the medical image (step ST21). Next, the text generation unit 23 generates a plurality of medical texts related to the medical image based on the attribute information (step ST22).

[0075] The multiple medical sentences can be generated by applying the beam search method described in, for example, "https: / / geekyisawesome.blogspot.com / 2016 / 10 / using-beam-serach-to-genrate-most.html" to the recurrent neural network 40 constituting the second learning model 23A. The beam search method is a method of searching for a word that appears next to a certain word, taking into account the occurrence probability of the word. In another embodiment, the sentence generation unit 23 applies the beam search method to the recurrent neural network 40 to generate multiple (e.g., five) medical sentences with words that have a high occurrence probability.

[0076] Next, the term identification unit 24 analyzes each of the medical sentences generated by the sentence generation unit 23 to identify terms expressing properties related to the structure of interest contained in each of the medical sentences (step ST23). Furthermore, the collation unit 25 collates the property information derived by the image analysis unit 22 with the terms identified by the term identification unit 24 for each of the medical sentences (step ST24), and selects one medical sentence from the medical sentences based on the result of the collation (step ST25). At this time, the collation unit 25 may select the medical sentence in which the property information and the terms most closely match. The collation unit 25 may select the medical sentence in which the order in which the property information is described in the sentence most closely matches the order of the property information derived by the image analysis unit 22. Then, the display control unit 26 displays the selected medical sentence and the collation result by the collation unit 25 on the display 15 (step ST26), and proceeds to the processing of step ST6 shown in FIG. 10.

[0077] In this way, in another embodiment, a plurality of medical sentences are generated, and the medical sentence with the most consistent terminology with the characteristic information is selected from the plurality of medical sentences. Therefore, the medical sentence with the least difference between the characteristic information and the included terminology can be presented to the radiologist together with the matching result. This reduces the frequency of correcting the medical sentence or regenerating the medical sentence, and as a result, reduces the burden on the radiologist who creates the radiology report.

[0078] In the above-mentioned other embodiment, one medical sentence is selected from the plurality of medical sentences, but two or more medical sentences may be selected. In this case, a predetermined number of medical sentences that are the top ones having the highest degree of matching between the condition information and the terms may be selected from the plurality of medical sentences. In addition, when a plurality of medical sentences are selected, it is preferable to display the plurality of medical sentences and the matching results for each of the plurality of medical sentences on the display 15, and allow the radiology interpreter to select the desired medical sentence.

[0079] In addition, in each of the above embodiments, a solid frame 77 is added to excessive terms in medical text 75 displayed in text display area 72 of display screen 70 shown in Fig. 7, and missing terms in medical text 75A displayed in text display area 72 of display screen 70A shown in Fig. 8 are displayed, and missing terms are added to dashed frame 79, but this is not limited to the above. As long as the excessive terms and missing terms can be distinguished and highlighted, any display mode can be used, such as changing the color of the text, adding a different type of dashed line, or changing the highlight color. In addition, the excess or shortage of terms may be notified by voice.

[0080] In addition, in each of the above embodiments, the collation unit 25 is equipped with the fourth learning model 25A, but is not limited to this. If it is possible to determine whether the characteristic information derived by the image analysis unit 22 matches the term indicating the characteristic included in the medical text identified by the term identification unit 24, any method other than the learning model can be applied.

[0081] In addition, in each of the above embodiments, the medical text 75 is recreated based on the selection of the automatic correction button 78A, but this is not limited to this. If the result of the collation by the collation unit 25 shows that the characteristic information derived by the image analysis unit 22 does not match the characteristics included in the medical text identified by the term identification unit 24, the text generation unit 23 may recreate the medical text 75 based on the result of the collation without waiting for the selection of the automatic correction button 78A.

[0082] In addition, in each of the above embodiments, the display control unit 26 displays both the medical text generated by the text generation unit 23 and the matching result by the matching unit 25, but this is not limited to this. Only the medical text may be displayed on the display screen 70 displayed on the display 15. In this case, for example, in response to an instruction to display the matching result from the input device 16, excessive or insufficient terms in the medical document may be highlighted, or an OK mark 80 may be displayed, as in each of the above embodiments.

[0083] In addition, in each of the above embodiments, the medical document creation support process such as a radiology report is performed by generating medical text using medical images of the lungs as the diagnostic target, but the diagnostic target is not limited to the lungs. In addition to the lungs, any part of the human body such as the heart, liver, brain, and limbs can be the diagnostic target. In this case, the image analysis unit 22, the text generation unit 23, and the term identification unit 24 are each provided with a learning model that performs analysis processing, text generation processing, and term identification processing according to the diagnostic target, and a learning model that performs analysis processing, text generation processing, and term identification processing according to the diagnostic target is selected to perform the medical text generation process.

[0084] In addition, in each of the above embodiments, the technology of the present disclosure is applied when creating an image reading report as a medical document, but it goes without saying that the technology of the present disclosure can also be applied when creating medical documents other than image reading reports, such as electronic medical records and diagnostic reports.

[0085] In addition, in each of the above embodiments, medical text is generated using medical images, but the present invention is not limited to this. It goes without saying that the technology disclosed herein can also be applied to the generation of text for any image other than medical images.

[0086] In each of the above embodiments, the following various processors can be used as the hardware structure of the processing unit that executes various processes, such as the image acquisition unit 21, image analysis unit 22, sentence generation unit 23, term identification unit 24, collation unit 25, and display control unit 26. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processes.

[0087] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0088] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as typified by System On Chip (SoC), etc. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0089] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0090] The following are supplementary claims to this application. (Additional note 1) at least one processor; The processor, By analyzing the image, property information indicative of properties of a structure of interest included in the image is derived; generating a sentence about the image based on the property information; Analyzing the text to identify terms contained in the text that describe properties related to the structure of interest; A document creation support device configured to match the property information with the term. (Additional note 2) 2. The document creation support device according to claim 1, wherein the processor is further configured to display the text on a display. (Additional note 3) 3. The document creation support device according to claim 2, wherein the processor is further configured to display a result of the comparison on a display. (Additional note 4) The document creation support device according to claim 3, wherein the processor is further configured to display the result of the comparison by highlighting the difference between the term and the property information in the sentence. (Additional note 5) 5. The document creation support device according to claim 1, wherein the processor is further configured to regenerate the sentence if the term and the property information differ in the sentence. (Additional note 6) 6. The document creation support device according to claim 1, wherein the processor is further configured to accept corrections to the text. (Additional note 7) The processor generates a plurality of sentences related to the image based on the property information; Analyzing each of the plurality of sentences to identify terms that are included in each of the plurality of sentences and that describe properties related to the structure of interest; Matching the attribute information with the terms for each of the plurality of sentences; 7. The document creation support device according to any one of claims 1 to 6, configured to select at least one sentence from the plurality of sentences based on a result of the comparison. (Additional note 8) 8. A document creation support device according to any one of claims 1 to 7, wherein the image is a medical image and the text is medical text relating to the structure of interest contained in the medical image. (Additional note 9) By analyzing the image, property information indicative of properties of a structure of interest included in the image is derived; generating a sentence about the image based on the property information; Analyzing the text to identify terms contained in the text that describe properties related to the structure of interest; A document creation support method for matching the property information with the terms. (Additional note 10) A step of deriving property information representing properties of a structure of interest included in an image by analyzing the image; generating a sentence about the image based on the property information; analyzing the text to identify terms contained in the text that represent properties related to the structure of interest; A document creation support program that causes a computer to execute a procedure for matching the property information with the terminology. [Explanation of symbols]

[0091] 1 Medical Information System 2. Modality 3. Image Reading Workstation 4 Departmental Workstations 5 Image Server 6. Image Database 7 Image interpretation report server 8. Radiographic report database 10 Network 11 CPU 12. Memory 13. Storage 14 Communication I / F 15 Display 16 Input Devices 20 Document creation support device 21 Image acquisition unit 22 Image Analysis Section 22A First Learning Model 23 Sentence generation section 23A Second Learning Model 24 Term identification part 24A The third learning model 25 Matching section 25A Fourth Learning Model 26 Display control section 30 Teacher Data 31 Abnormal shadow 32 Medical Imaging 33 Properties information 40 Recurrent Neural Networks 41 Encoder 42 Decoder 50 Teacher Data 51 Medical Texts 52 Terms expressing characteristics 60,65 Teacher data 61,66 Properties information 62,67 Terms describing characteristics 63,68 Surplus / deficiency information 70,70A,70B display screen 71 Image display area 72 Text display area 73 Abnormal shadow candidate 74 Rectangular area 75, 75A, 75B Medical Documents 76 Properties information 77 Solid frame 78A Auto-correction button 78B Manual correction button 78C Confirm button 79 Dashed Frame 80 OK mark 81 Arrow SL1 slice image

Claims

1. At least one processor; The processor, By analyzing the image, property information indicative of properties of a structure of interest included in the image is derived; generating a plurality of sentences relating to the image based on the property information; Analyzing the plurality of sentences to identify terms that are included in each of the plurality of sentences and that express properties related to the structure of interest; Matching the attribute information with the terms for each of the plurality of sentences; The document creation support device is configured to select, from the plurality of sentences, at least one sentence that ranks highest in degree of match between the property information and the term.

2. 2. The document creation support device according to claim 1, wherein the processor is further configured to display the selected sentence on a display.

3. 3. The document creation support device according to claim 2, wherein the processor is further configured to display a result of the matching for the selected sentence on a display.

4. The document creation support device according to claim 3 , wherein the processor is further configured to, when the term and the property information in the selected sentence differ, display the result of the comparison by highlighting the difference.

5. The document creation support device according to claim 1 , wherein the processor is further configured to regenerate the selected sentence if the term and the property information differ in the selected sentence.

6. 6. The document creation support apparatus according to claim 1, wherein the processor is further configured to accept corrections to the selected sentence.

7. The document creation support device according to claim 1 , wherein the image is a medical image, and the text is medical text relating to the structure of interest included in the medical image.

8. A step of deriving property information representing properties of a structure of interest included in an image by analyzing the image; generating a plurality of sentences relating to the image based on the property information; a step of identifying a term that represents a property related to the structure of interest contained in each of the plurality of sentences by analyzing the plurality of sentences; matching the property information with the terms for each of the plurality of sentences; and selecting from the plurality of sentences at least one sentence that ranks highest in degree of match between the property information and the term.

Images