Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, information processing system, information processing method, and information processing program, which allow for assisting in generation of medical documents.SOLUTION: An information processing device 10 provided herein comprises at least one processor configured to acquire multiple pieces of element information pertaining to an image, generate a plan defining an element description order in a sentence describing elements corresponding to the multiple pieces of element information, and generate a sentence according to the plan.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, image diagnosis has been performed using medical images obtained by imaging devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices. Furthermore, medical images have been analyzed by CAD (Computer Aided Detection / Diagnosis) using classifiers trained by deep learning or the like to detect and / or diagnose regions of interest, including structures and lesions, contained in the medical images. The medical images and the results of the CAD analysis are transmitted to a terminal of a medical professional, such as a radiologist, who interprets the medical images. The medical professional, such as a radiologist, uses his or her own terminal to refer to the medical images and the analysis results, interprets the medical images, and creates a radiology report.

[0003] Furthermore, various methods for supporting the creation of medical documents such as radiology reports have been proposed to reduce the workload of radiologists performing radiology interpretation work. For example, Patent Document 1 discloses a technology for creating medical documents such as radiology reports based on keywords entered by a radiologist and the results of analyzing medical images. The technology described in Patent Document 1 uses a recurrent neural network trained to generate text from input characters to create text to be included in the radiology report. For example, Patent Document 2 discloses a technology for creating, for annotated medical images, fixed phrases associated in advance with each type of annotation as the findings of the medical images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-153250 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-257579 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, with the increasing performance of imaging devices, the amount of information in analytical results obtained from medical images has been increasing, and the amount of text included in medical documents such as radiology reports has also been increasing.To make the large amount of text easier to read, medical documents may have rules regarding the order in which analytical results obtained from medical images are written, such as agreements within medical institutions or user preferences.

[0006] For example, when describing an abnormal shadow in a medical image, it may be desirable to describe the overall characteristics of the abnormal shadow first, followed by the characteristics of its periphery and interior. Also, for example, it may be desirable to describe malignant findings first, followed by benign findings. Furthermore, when describing the results of a comparison with a previous medical image, it may be desirable to describe changed areas first, followed by unchanged areas.

[0007] However, when attempting to generate text containing a large amount of complex information using conventional techniques, there are cases where the text is not written in the desired order, information is omitted, or the text becomes redundant. Therefore, there is a demand for a technique that can generate text in a proper order and with a proper degree of information coverage for text to be written in medical documents, etc.

[0008] The present disclosure provides an information processing device, an information processing method, and an information processing program that can support the creation of medical documents. [Means for solving the problem]

[0009] A first aspect of the present disclosure is an information processing device including at least one processor, the processor acquiring a plurality of pieces of element information related to an image, and generating a first trained model that is trained in advance to use the plurality of pieces of element information as input and output as a plan, the first trained model including at least one piece of element information arranged in accordance with the order in which elements corresponding to the plurality of pieces of element information are described in a sentence in which the elements are described. By inputting multiple element information into and receiving a plan from the first trained model, A second trained model that has been trained to generate a sentence in which elements corresponding to the element information contained in the plan are written in the order in which they are written, with the plan as input and the sentence as output. By inputting the plan into the and receiving sentences from the second trained model, Generate.

[0010] In the first aspect, the processor may divide a plurality of pieces of element information into groups and generate a plan in which the order of description is determined for each group.

[0011] In the first aspect, the processor may accept specification of at least one rule from among a plurality of different predetermined rules for dividing a plurality of pieces of element information into groups, and divide the plurality of pieces of element information into groups in accordance with the specified rule.

[0012] In the first aspect, the processor may obtain a plurality of pieces of element information relating to each of a plurality of regions of interest included in the image, and divide the plurality of pieces of element information into a plurality of groups corresponding to each of the plurality of regions of interest.

[0013] In the first aspect, the plurality of pieces of element information may relate to each of the plurality of images, and the processor may generate a sentence that collectively describes elements corresponding to the plurality of pieces of element information relating to each of the plurality of images.

[0014] In the first aspect, the processor may acquire element information indicating the time at which the image was captured, and generate a plan that determines the order in which elements corresponding to the related element information are written based on the time at which the image was captured indicated by the element information.

[0015] In the first aspect, the importance of each element information is predetermined, and the processor may generate a plan that specifies that elements corresponding to element information among the plurality of element information whose importance is lower than a predetermined threshold value will not be described in the text.

[0016] In the first aspect, the importance of each element information is predetermined, and the processor may generate a plan that specifies that elements corresponding to element information with a relatively low importance among the multiple element information will not be included in the text.

[0017] In the first aspect, the processor may receive a specification of the degree of conciseness of the sentence, and change the number of elements not to be described in the sentence according to the degree of conciseness.

[0019] In the first aspect, the first trained model may be trained using, as training data, a set of element information corresponding to elements contained in a previously generated sentence and a plan that determines the order in which the elements are written in the sentence.

[0020] In the first aspect, the processor may generate multiple different plan candidates for multiple element information, generate sentences for each plan candidate, evaluate each sentence, and determine the plan candidate to adopt based on the results of the evaluation.

[0021] In the first aspect, the processor may perform the evaluation based on at least one of the order in which elements included in the generated sentence are described and the degree of coverage of the elements.

[0022] In the first aspect, the processor may generate a plurality of different plan candidates for a plurality of pieces of element information, and may receive designation of a plan candidate to be adopted from among the plurality of different plan candidates.

[0023] In the first aspect, the processor may acquire an image and generate the element information based on the acquired image.

[0024] In the first aspect, the device may further include an input unit, and the processor may generate the element information based on information input via the input unit.

[0025] In the first aspect, the element information may be information indicating at least one of the name, characteristics, measurement values ​​and position of the region of interest contained in the image, and the imaging method, imaging conditions and imaging date and time for capturing the image.

[0026] In the first aspect, the image is a medical image, and the element information is information indicating at least one of the name, characteristics, location, and suspected disease name of the area of ​​interest contained in the medical image, as well as the imaging method, imaging conditions, and imaging date and time for capturing the medical image, and the area of ​​interest may be at least one of the area of ​​a structure contained in the medical image and the area of ​​an abnormal shadow contained in the medical image.

[0027] A second aspect of the present disclosure is an information processing method, comprising: acquiring a plurality of pieces of element information related to an image; and using a first trained model that has been trained in advance to input a plan including at least one of a plurality of pieces of element information arranged in accordance with the order in which elements corresponding to the plurality of pieces of element information are described in a sentence in which the elements are described; and outputting the plan. By inputting multiple element information into and receiving a plan from the first trained model, A second trained model that has been trained to generate a sentence in which elements corresponding to the element information contained in the plan are written in the order in which they are written, with the plan as input and the sentence as output. By inputting the plan into the and receiving sentences from the second trained model, The generating process is performed by a computer.

[0028] A third aspect of the present disclosure is an information processing program, comprising: a first trained model that is pre-trained to acquire a plurality of pieces of element information related to an image, and to output a plan including at least one of a plurality of pieces of element information arranged in accordance with the order in which elements corresponding to the plurality of pieces of element information are described in a sentence in which the elements are described; and By inputting multiple element information into and receiving a plan from the first trained model, A second trained model that has been trained to generate a sentence in which elements corresponding to the element information contained in the plan are written in the order in which they are written, with the plan as input and the sentence as output. By inputting the plan into the and receiving sentences from the second trained model,It is intended to cause a computer to execute the generation process. [Effects of the Invention]

[0029] According to the above aspects, the information processing device, information processing method, and information processing program of the present disclosure can assist in creating medical documents. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 3] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a medical image. [Figure 5] FIG. 10 is a diagram illustrating an example of element information. [Figure 6] FIG. 10 is a diagram illustrating an example of a sentence. [Figure 7] FIG. 2 is a diagram for explaining processing according to the first embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a plan. [Figure 9] FIG. 10 is a diagram illustrating an example of a screen displayed on a display. [Figure 10] 10 is a flowchart showing an example of a first information process. [Figure 11] FIG. 10 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 12] FIG. 10 is a diagram for explaining processing according to the second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of plan candidates. [Figure 14] FIG. 10 is a diagram illustrating an example of plan candidates. [Figure 15] FIG. 10 is a diagram illustrating an example of plan candidates. [Figure 16] 10 is a flowchart showing an example of a second information process. [Figure 17] FIG. 10 is a diagram for explaining a process according to a conventional method. [Figure 18] FIG. 10 is a diagram illustrating an example of a sentence. [Figure 19] 1 shows the evaluation results of the methods according to the first and second embodiments and the conventional method. [Figure 20] FIG. 10 is a diagram illustrating an example of a plan. [Figure 21] FIG. 10 is a diagram illustrating an example of a plan. [Figure 22] FIG. 10 is a diagram illustrating an example of a plan. [Figure 23] FIG. 10 is a diagram illustrating an example of a plan. [Figure 24] FIG. 10 is a diagram illustrating an example of a screen displayed on a display. [Figure 25] FIG. 10 is a diagram illustrating an example of a screen displayed on a display. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, each embodiment of the present disclosure will be described with reference to the drawings.

[0032] [First embodiment] First, the configuration of an information processing system 1 to which an information processing device according to the present disclosure is applied will be described. Fig. 1 is a diagram showing a schematic configuration of the information processing system 1. The information processing system 1 shown in Fig. 1 photographs the region to be examined of a subject and stores the medical images acquired by photographing, based on an examination order from a doctor of a medical department using a known ordering system. It also allows a radiologist to interpret the medical images and create an interpretation report, and allows the doctor of the requesting medical department to view the interpretation report.

[0033] 1, the information processing system 1 includes an imaging device 2, an interpretation WS (Workstation) 3 which is an interpretation terminal, a medical treatment WS 4, an image server 5, an image DB (DataBase) 6, a report server 7, and a report DB 8. The imaging device 2, the interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 are connected via a wired or wireless network 9 in a state where they can communicate with each other.

[0034] Each device is a computer installed with an application program that causes the device to function as a component of the information processing system 1. The application program may be recorded on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) and distributed, and then installed on the computer from the recording medium. Alternatively, the application program may be stored in a storage device or network storage of a server computer connected to the network 9 in an externally accessible state, and downloaded to the computer upon request and installed.

[0035] The imaging device 2 is a device (modality) that captures an image of a diagnostic target region of a subject, thereby generating a medical image representing the diagnostic target region. Specifically, it is a plain X-ray imaging device, a CT device, an MRI device, a PET (Positron Emission Tomography) device, etc. The medical image generated by the imaging device 2 is transmitted to the image server 5 and stored in the image DB 6.

[0036] The image interpretation WS3 is a computer used by a medical professional, such as a radiologist, to interpret medical images and create image interpretation reports, and includes the information processing device 10 according to this embodiment. The image interpretation WS3 issues requests for viewing medical images to the image server 5, performs various image processing on medical images received from the image server 5, displays the medical images, and accepts input of text related to the medical images. The image interpretation WS3 also performs analysis processing on medical images, supports the creation of image interpretation reports based on the analysis results, requests the report server 7 to register and view image interpretation reports, and displays image interpretation reports received from the report server 7. These processes are performed by the image interpretation WS3 executing software programs for each process.

[0037] The medical WS4 is a computer used by medical personnel such as doctors in a medical department to perform detailed observations of medical images, view interpretation reports, and create electronic medical records, and is composed of a processing device, a display device such as a monitor, and input devices such as a keyboard and a mouse. The medical WS4 sends requests to the image server 5 to view medical images, displays medical images received from the image server 5, requests to the report server 7 to view interpretation reports, and displays interpretation reports received from the report server 7. These processes are performed by the medical WS4 executing software programs for each process.

[0038] The image server 5 is a general-purpose computer installed with a software program that provides the functions of a database management system (DBMS). The image server 5 is connected to an image DB 6. The connection between the image server 5 and the image DB 6 is not particularly limited, and may be via a data bus or a network such as a NAS (Network Attached Storage) or a SAN (Storage Area Network).

[0039] The image DB 6 is realized by a storage medium such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. In the image DB 6, medical images acquired by the imaging device 2 and accompanying information attached to the medical images are registered in association with each other.

[0040] The incidental information may include, for example, identification information such as an image ID (identification) for identifying a medical image, a cross-sectional ID assigned to each cross-sectional image included in the medical image, a subject ID for identifying the subject, and an examination ID for identifying the examination. The incidental information may also include, for example, information related to imaging, such as the imaging method, imaging conditions, and imaging date and time for capturing the medical image. The "imaging method" and "imaging conditions" refer to, for example, the type of imaging device 2, the imaging region, the imaging protocol, the imaging sequence, the imaging technique, whether or not a contrast agent was used, etc. The incidental information may also include information related to the subject, such as the subject's name, age, and gender.

[0041] Furthermore, when the image server 5 receives a request to register a medical image from the imaging device 2, it converts the medical image into a database format and registers it in the image DB 6. Furthermore, when the image server 5 receives a request to view from the image interpretation WS 3 and the medical treatment WS 4, it searches for medical images registered in the image DB 6 and transmits the searched medical images to the image interpretation WS 3 and the medical treatment WS 4 that have sent the request to view.

[0042] The report server 7 is a general-purpose computer on which a software program that provides the functions of a database management system is installed. The report server 7 is connected to a report DB 8. The connection between the report server 7 and the report DB 8 is not particularly limited, and they may be connected by a data bus or via a network such as a NAS or SAN.

[0043] The report DB 8 is realized by a storage medium such as an HDD, an SSD, a flash memory, etc. The report DB 8 registers the interpretation reports created in the interpretation WS 3.

[0044] Furthermore, when the report server 7 receives a request to register an interpretation report from the image interpretation WS 3, it formats the interpretation report for a database and registers it in the report DB 8. Furthermore, when the report server 7 receives a request to view an interpretation report from the image interpretation WS 3 and the medical care WS 4, it searches for the interpretation report registered in the report DB 8 and transmits the searched interpretation report to the image interpretation WS 3 and the medical care WS 4 that have made the view request.

[0045] The network 9 is, for example, a network such as a LAN (Local Area Network) or a WAN (Wide Area Network). The imaging device 2, the interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 included in the information processing system 1 may be located in the same medical institution, or in different medical institutions. The number of each device of the imaging device 2, the interpretation WS 3, the medical treatment WS 4, the image server 5, the image DB 6, the report server 7, and the report DB 8 is not limited to the number shown in Fig. 1, and each device may be composed of multiple devices having similar functions.

[0046] Next, we will explain the information processing device 10 according to this embodiment. The information processing device 10 has a function of supporting the creation of medical documents such as radiology reports based on medical images captured by the imaging device 2. As described above, the information processing device 10 is included in the radiology WS3.

[0047] First, an example of the hardware configuration of an information processing device 10 according to this embodiment will be described with reference to Fig. 2. As shown in Fig. 2, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a keyboard and a mouse, and a network I / F (Interface) 26. The network I / F 26 is connected to a network 9 and performs wired or wireless communication. The CPU 21, the storage unit 22, the memory 23, the display 24, the input unit 25, and the network I / F 26 are connected via a bus 28 such as a system bus and a control bus so as to be able to exchange various information with one another.

[0048] The storage unit 22 is realized by a storage medium such as an HDD, an SSD, or a flash memory. The storage unit 22 stores an information processing program 27 for the information processing device 10. The CPU 21 reads the information processing program 27 from the storage unit 22, loads it into the memory 23, and executes the loaded information processing program 27. The CPU 21 is an example of a processor of the present disclosure. For example, a personal computer, a server computer, a smartphone, a tablet terminal, a wearable terminal, etc. can be used as the information processing device 10, as appropriate.

[0049] Next, an example of the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the information processing device 10 includes an acquisition unit 30, a first generation unit 32, a second generation unit 34, and a control unit 36. The first generation unit 32 may include a plan generation model M1, and the second generation unit 34 may include a sentence generation model M2 (details will be described later). The CPU 21 executes the information processing program 27, causing the CPU 21 to function as the acquisition unit 30, the first generation unit 32, the second generation unit 34, and the control unit 36.

[0050] The acquisition unit 30 acquires, from the image server 5, a medical image for which an interpretation report is to be created. FIG. 4 shows, as an example of a medical image, a medical image 50X obtained by capturing lungs using a CT device. The medical image 50X includes an abnormal shadow N indicating a nodule. Hereinafter, at least one of the region of a structure (e.g., organs, tissues, etc., such as the lungs and trachea) included in the medical image and the region of an abnormal shadow (e.g., a shadow due to a lesion such as a nodule) included in the medical image will be referred to as a region of interest. Note that one medical image may include multiple regions of interest. The medical image is an example of an image of the present disclosure.

[0051] The acquiring unit 30 also acquires a plurality of pieces of element information related to the medical image acquired from the image server 5. Fig. 5 shows, as an example of element information, a plurality of pieces of element information 52X related to an abnormal shadow N included in a medical image 50X (see Fig. 4). As shown in Fig. 5, the element information may be information indicating at least one of elements such as the name (type), characteristics, measurement value, position, and suspected disease name (including a negative or positive evaluation result) related to a region of interest included in the medical image.

[0052] Examples of names (types) include names of structures such as "lung," "trachea," and "pleura," as well as names of abnormal shadows such as "nodule," "cavity," and "calcification." Characteristics primarily refer to the characteristics of abnormal shadows. For example, in the case of nodules, these include absorption values ​​such as "solid" and "ground-glass," margin shapes such as "clear / indistinct," "smooth / irregular," "spicule," "lobulated," and "serrated," and findings indicating the overall shape such as "near-circular" and "irregular." Other examples include qualitative findings indicating the size and quantity of abnormal shadows (such as "large / small" and "single / multiple"), as well as findings regarding the presence or absence of contrast enhancement and washout.

[0053] Measurement values ​​are values ​​that can be quantitatively measured from medical images, such as the major axis, CT value in HU units, and the number of regions of interest (if multiple regions of interest exist), as well as the distance between the regions of interest. Position refers to the position of a region of interest in the medical image or its positional relationship with other regions of interest, such as "inside," "edge," and "periphery." The estimated disease name is the evaluation result estimated by the acquisition unit 30 based on the abnormal shadow, such as disease names like "cancer" and "inflammation," and evaluation results like "negative / positive" for each characteristic. In Figure 5, characteristics evaluated as negative are marked with [-], and characteristics evaluated as positive are left blank.

[0054] For example, the acquisition unit 30 may generate the above-described element information based on the acquired medical image by using CAD. Specifically, the acquisition unit 30 extracts a region of interest included in the medical image. For example, the extraction of the region of interest may use a trained model such as a convolutional neural network (CNN) that is trained in advance to input a medical image and output a region of interest extracted from the medical image. Furthermore, the acquisition unit 30 may extract, as the region of interest, a region in the medical image specified by the user via the input unit 25.

[0055] Thereafter, the acquiring unit 30 generates element information related to the region of interest extracted from the medical image. For generating the element information by the acquiring unit 30, for example, a trained model such as CNN that is trained in advance to use the region of interest in the medical image as input and the element information related to the region of interest as output may be used.

[0056] Furthermore, for example, the acquiring unit 30 may generate element information based on information input via the input unit 25. Specifically, the acquiring unit 30 may generate element information based on a keyword input by the user via the input unit 25. Furthermore, for example, the acquiring unit 30 may present candidates for element information on the display 24 and accept designation of element information by the user.

[0057] Furthermore, as described above, incidental information including information related to imaging is attached to each medical image at the time of registration in the image DB 6. Therefore, for example, the acquisition unit 30 may generate, as element information, information indicating at least one of the imaging method, imaging conditions, and imaging date and time related to imaging of the medical image, based on the incidental information attached to the medical image acquired from the image server 5.

[0058] For example, the acquiring unit 30 may acquire element information that has been generated in advance by an external device having a function of generating element information based on a medical image using the above-mentioned CAD from the external device. For example, the acquiring unit 30 may acquire information contained in test orders and electronic medical records, information indicating various test results such as blood tests and infectious disease tests, and information indicating the results of health checkups from an external device such as the medical WS 4, and generate the information as element information as appropriate.

[0059] In medical documents such as radiology reports, the order in which elements corresponding to multiple pieces of information about medical images are described may be determined based on rules, such as internal agreements within medical institutions or user preferences, to facilitate reader understanding of the content of the text. For example, when describing an abnormal shadow such as a nodule, it may be desirable to describe findings such as the location, size, and overall shape first, followed by detailed findings about the margins and interior. It may also be desirable to describe malignant findings first and benign findings second. It may also be desirable to describe the results of a comparison between past and present medical images of the same subject, describing changed areas first and unchanged areas second.

[0060] 6 shows an example of text 56X in which elements corresponding to a plurality of pieces of element information 52X (see FIG. 5) regarding an abnormal shadow N (nodule) contained in a medical image 50X (see FIG. 4) are written in an appropriate order. Text 56X consists of four paragraphs, and the elements are written in order from the beginning to the end, namely, the overall characteristics of the nodule, the characteristics of its margins, the characteristics of its interior, and the relationship with surrounding tissues used to determine the extent of the nodule's progression.

[0061] The first generation unit 32 and the second generation unit 34 according to this embodiment generate a sentence in which elements corresponding to a plurality of pieces of element information acquired and / or generated by the acquisition unit 30 are written in an appropriate order, as shown in Fig. 6. The functions of the first generation unit 32 and the second generation unit 34 will be described below with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram showing the order of processing by the first generation unit 32 and the second generation unit 34 according to this embodiment.

[0062] First, the first generating unit 32 generates a plan that defines the order in which elements corresponding to the plurality of pieces of element information acquired and / or generated by the acquiring unit 30 will be written in a sentence in which the elements are written. The plan defines the paragraph structure of the entire sentence (i.e., the number and order of paragraphs) and in which paragraph each element corresponding to the plurality of pieces of element information will be written. Note that the plan defines at least the paragraph structure and in which paragraph each element will be written, and the order in which each element will be written in a paragraph does not necessarily have to be defined.

[0063] In generating the plan, the first generation unit 32 may divide the plurality of pieces of element information acquired and / or generated by the acquisition unit 30 into groups, and generate a plan in which the order of description is determined for each group. That is, one group may be regarded as one paragraph, and by dividing the plurality of pieces of element information into groups, it may be possible to determine in which paragraph each element corresponding to the plurality of pieces of element information is to be described, and by determining the order of each group, it may be possible to determine the order of description of each paragraph in the entire sentence.

[0064] 8 shows a plan 54X as an example of a plan, and a sentence 56X generated based on the plan 54X. The plan 54X corresponds to a plurality of pieces of element information 52X (see FIG. 5) related to a medical image 50X (see FIG. 4). In FIG. 8, the plurality of pieces of element information 52X related to "nodule" are divided into four groups: those indicating the overall characteristics (group 1), those indicating the characteristics of the periphery (group 2), those indicating the characteristics of the interior (group 3), and those indicating the relationship with surrounding tissues (group 4). Groups 1 to 4 correspond to paragraphs arranged in ascending order from the beginning of the sentence.

[0065] For generating a plan by the first generation unit 32, a plan generation model M1 such as a CNN or an RNN (Recurrent Neural Network) that has been trained in advance to use element information as input and a plan as output, as shown in Fig. 7, may be used. The plan generation model M1 is a model that is trained using, as training data, a set of element information corresponding to elements included in a sentence generated in the past and a plan in which the order in which the elements are written in the sentence is determined. The plan generation model M1 is an example of a first trained model of the present disclosure.

[0066] The plans used as training data for the plan generation model M1 reflect predetermined rules such as the agreements made within the medical institution regarding the order in which elements in a sentence should be written, the user's preferences, etc. By learning using such plans as training data, the plan generation model M1 can group the input element information according to the rules and generate plans in which the order in which each group should be written is determined.

[0067] Next, the second generation unit 34 generates a sentence based on the plan generated by the first generation unit 32. Specifically, as shown in Fig. 8, the second generation unit 34 generates a paragraph including at least one sentence for each group defined in the plan 54X, and finally compiles the paragraphs to generate one sentence 56X.

[0068] For generating sentences by the second generation unit 34, a sentence generation model M2 such as a CNN or RNN that has been trained in advance to take a plan as input and a sentence as output may be used, as shown in Fig. 7. The sentence generation model M2 is a model that is trained using pairs of previously generated sentences and plans related to those sentences as training data. The sentence generation model M2 is an example of a second trained model of the present disclosure.

[0069] The control unit 36 ​​controls the display of the text generated by the second generation unit 34 on the display 24. Fig. 9 shows an example of a screen D1 on which the text is displayed, which is displayed on the display 24 by the control unit 36. The screen D1 includes a medical image 50X (see Fig. 4), a plurality of pieces of element information 52X (see Fig. 5) acquired and / or generated by the acquisition unit 30, and a text 56X generated by the second generation unit 34.

[0070] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Fig. 10. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the first information processing shown in Fig. 10. The first information processing is executed, for example, when a user issues an instruction to start execution via the input unit 25.

[0071] In step S10, the acquisition unit 30 acquires a medical image from the image server 5. In step S12, the acquisition unit 30 acquires and / or generates a plurality of pieces of element information related to the medical image acquired in step S10. Specifically, the acquisition unit 30 may generate a plurality of pieces of element information based on the medical image acquired in step S10, or may generate the element information based on information input by a user via the input unit 25 and information acquired from an external device. The acquisition unit 30 may also acquire the element information from an external device.

[0072] In step S14, the first generation unit 32 generates a plan that defines the order in which elements corresponding to the plurality of element information acquired and / or generated in step S12 will be written in a sentence in which the elements are written. In step S16, the second generation unit 34 generates a sentence based on the plan generated in step S14. In step S18, the control unit 36 ​​causes the display 24 to display a screen including the sentence generated in step S16, and ends this first information processing.

[0073] As described above, the information processing device 10 according to one aspect of the present disclosure includes at least one processor, which acquires a plurality of pieces of element information related to an image, generates a plan that defines the order in which elements corresponding to the plurality of pieces of element information are to be described in a sentence, and generates the sentence based on the plan. In other words, the information processing device 10 according to this embodiment can generate a sentence in which the order and degree of coverage of elements are appropriate, thereby assisting in the creation of medical documents.

[0074] [Second embodiment] In addition to the same functions as those of the first embodiment, the information processing device 10 according to the second embodiment generates multiple plan candidates for multiple pieces of element information, evaluates each plan candidate, and selects the most appropriate plan candidate, thereby enabling the generation of more appropriate sentences. The information processing device 10 according to the second embodiment will be described below, but descriptions of the same configurations and functions as those of the first embodiment will be omitted where appropriate.

[0075] An example of the functional configuration of the information processing device 10 according to this embodiment will be described with reference to Fig. 11. As shown in Fig. 11, the information processing device 10 includes an acquisition unit 30, a first generation unit 32, a second generation unit 34, a control unit 36, and an evaluation unit 38. The first generation unit 32 may include a plan generation model M1, and the second generation unit 34 may include a sentence generation model M2. The evaluation unit 38 may include a sentence evaluation model M3 (details will be described later). When the CPU 21 executes the information processing program 27, the CPU 21 functions as the acquisition unit 30, the first generation unit 32, the second generation unit 34, the control unit 36, and the evaluation unit 38.

[0076] The acquisition unit 30 acquires medical images for which an interpretation report is to be created from the image server 5. The acquisition unit 30 also acquires and / or generates a plurality of element information related to the medical images acquired from the image server 5. The function of the acquisition unit 30 is the same as that in the first embodiment, and therefore a description thereof will be omitted.

[0077] The functions of the first generation unit 32, the second generation unit 34, and the evaluation unit 38 according to this embodiment will be described below with reference to Figures 12 to 15. Figure 12 is a diagram showing the order of processing by the first generation unit 32, the second generation unit 34, and the evaluation unit 38 according to this embodiment. In the following explanation, a form will be described in which the first generation unit 32, the second generation unit 34, and the evaluation unit 38 generate plan candidates, generate sentences, and evaluate each paragraph.

[0078] First, the first generation unit 32 generates a plurality of different plan candidates for the plurality of element information acquired and / or generated by the acquisition unit 30. That is, the first generation unit 32 generates a plurality of variations of plan candidates without changing the plurality of element information themselves that are the subject of plan candidate generation. The variations of the plan candidates may be, for example, different numbers and types of groups in the plans, or the number and types of groups may be the same but different element information may be assigned to each group.

[0079] As an example of multiple different plan candidates, plan candidate 64A is shown in FIG. 13, plan candidate 64B is shown in FIG. 14, and plan candidate 64C is shown in FIG. 15. Plan candidates 64A to 64C each correspond to multiple pieces of element information 52X (see FIG. 5) related to medical image 50X (see FIG. 4). FIGS. 13 to 15 show, as plan candidates 64A to 64C, plan candidates for a portion corresponding to one group at the beginning of a sentence (i.e., one paragraph at the beginning of a sentence) for multiple pieces of element information 52X related to "nodule." In this way, first generation unit 32 first generates multiple plan candidates for a portion corresponding to one group (i.e., one paragraph).

[0080] The method of generating plan candidates by the first generating unit 32 is the same as the method of generating plans in the first embodiment, and therefore description thereof will be omitted. For example, the generation of plan candidates by the first generating unit 32 may use a plan generation model M1 as shown in FIG.

[0081] Next, the second generation unit 34 generates a sentence for each plan candidate generated by the first generation unit 32. Specifically, the second generation unit 34 generates a sentence for one paragraph (hereinafter referred to as a "sentence candidate") based on the plan candidates for a portion corresponding to one group generated by the first generation unit 32. As an example of the sentence candidates generated by the second generation unit 34, FIGS. 13 to 15 respectively show sentence candidates 66A to 66C for one paragraph generated based on plan candidates 64A to 64C for one group.

[0082] The method of generating sentence candidates by the second generation unit 34 is the same as the method of generating sentences in the first embodiment, and therefore the description thereof will be omitted. For example, the generation of sentence candidates by the second generation unit 34 may use a sentence generation model M2 as shown in FIG.

[0083] Next, the evaluation unit 38 performs an evaluation for each sentence (sentence candidate) generated by the second generation unit 34. Specifically, the evaluation unit 38 performs the evaluation based on at least one of the order of description and the degree of comprehensiveness of elements included in the sentence (sentence candidate) generated by the second generation unit 34. Evaluation based on the order of description means evaluating whether the elements included in the sentence are written in an appropriate order. Evaluation based on the degree of comprehensiveness means evaluating whether the elements that should be written in the sentence are written appropriately, without any excess or deficiency. In this embodiment, since plan candidates and sentences (sentence candidates) are generated for each paragraph, the evaluation unit 38 can evaluate the order of description and the degree of comprehensiveness by evaluating whether the elements that should be written in the paragraph currently being evaluated are written appropriately, without any excess or deficiency.

[0084] As an example of the results of the evaluation by the evaluation unit 38, FIGS. 13 to 15 show evaluation scores 68A to 68C for sentence candidates 66A to 66C, respectively. The evaluation scores 68A to 68C are determined so that the minimum value is 0 and the maximum value is 100, with the higher the evaluation, the larger the value. Sentence candidate 66A has a high evaluation score 68A because elements indicating the overall characteristics of the nodule are appropriately described. On the other hand, sentence candidate 66B has an excessive number of elements indicating the internal characteristics of the nodule ("hollow [-]"), so evaluation score 68B is lower than evaluation score 68A. Furthermore, sentence candidate 66C has an element indicating the peripheral characteristics of the nodule rather than an element indicating the overall characteristics of the nodule, so evaluation score 68C is the lowest.

[0085] For the evaluation of a sentence (sentence candidate) by the evaluation unit 38, a sentence evaluation model M3 such as BERT (Bidirectional Encoder Representations from Transformers) may be used, which is trained in advance to take a sentence (sentence candidate) as input and an evaluation score as output, as shown in Fig. 12. The sentence evaluation model M3 is a model that trains sentences generated by users such as doctors as correct sentences with appropriate order and degree of coverage. In other words, the sentence evaluation model M3 is a model that calculates an evaluation score according to the similarity between the input sentence and a correct sentence that has been trained in advance, in terms of order and degree of coverage.

[0086] Furthermore, the evaluation unit 38 determines the plan candidate to be adopted based on the results of the evaluation of the sentences. For example, in the examples of Figures 13 to 15, the evaluation unit 38 selects the plan candidate 64A corresponding to the sentence candidate 66A that has been assigned the highest evaluation score 68A.

[0087] The first generation unit 32, the second generation unit 34, and the evaluation unit 38 repeat the above process for each paragraph to determine candidate plans to be adopted for all paragraphs. This allows a plan with an appropriate order of description and degree of coverage to be selected for each paragraph. When evaluating a sentence (sentence candidate), the evaluation unit 38 may also take into account the results of evaluation of the previous paragraph that has already been evaluated.

[0088] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Fig. 16. In the information processing device 10, the CPU 21 executes the information processing program 27, thereby executing the second information processing shown in Fig. 16. The second information processing is executed, for example, when a command to start execution is given by the user via the input unit 25.

[0089] In step S40, the acquisition unit 30 acquires a medical image from the image server 5. In step S42, the acquisition unit 30 acquires and / or generates a plurality of pieces of element information related to the medical image acquired in step S40. Specifically, the acquisition unit 30 may generate a plurality of pieces of element information based on the medical image acquired in step S40, or may generate the element information based on information input by a user via the input unit 25 and information acquired from an external device. The acquisition unit 30 may also acquire the element information from an external device.

[0090] In step S44, the first generation unit 32 generates a plurality of plan candidates corresponding to one paragraph of text in which elements corresponding to the plurality of element information acquired and / or generated in step S42 are described. In step S46, the second generation unit 34 generates a text (sentence candidate) for each of the plurality of plan candidates generated in step S44. In step S48, the evaluation unit 38 evaluates each of the sentences (sentence candidate) generated in step S46. In step S50, the evaluation unit 38 determines the plan candidate to adopt based on the result of the evaluation in step S48.

[0091] In step S52, the evaluation unit 38 determines whether or not candidates for the plan to be adopted have been determined for all paragraphs of the text in which elements corresponding to the plurality of element information acquired and / or generated in step S42 are described. If the determination of candidates for the plan to be adopted for all paragraphs has not been completed (i.e., if step S52 is N), the processes of steps S44 to S50 are repeated for paragraphs for which no candidates for the plan have yet been generated.

[0092] On the other hand, if the candidate plans to be adopted for all paragraphs have been determined (i.e., if step S52 is Y), the process proceeds to step S54. In step S54, the control unit 36 ​​causes the display 24 to display a screen including sentences generated based on the candidate plans for each paragraph determined in step S50, and ends this second information processing.

[0093] As described above, the information processing device 10 according to one aspect of the present disclosure includes at least one processor, which generates a plurality of different plan candidates for a plurality of pieces of element information, generates a sentence for each plan candidate, evaluates each sentence, and determines the plan candidate to adopt based on the evaluation results. In other words, the information processing device 10 according to this embodiment can generate sentences with more appropriate description order and coverage, thereby supporting the creation of medical documents.

[0094] In the second embodiment, the evaluation unit 38 determines which plan candidate to adopt from among the multiple plan candidates generated by the first generation unit 32. However, the present invention is not limited to this. For example, the control unit 36 ​​may receive a designation of a plan candidate to adopt from among the multiple different plan candidates generated by the first generation unit 32. Specifically, the control unit 36 ​​may control the display 24 to display the multiple different plan candidates generated by the first generation unit 32, and determine the plan candidate to adopt based on a designation by the user via the input unit 25. In this case, the information processing device 10 may omit the function of the evaluation unit 38.

[0095] [Comparison with conventional methods] A comparison between the methods according to the first and second embodiments and a conventional method will be described. Fig. 17 is a diagram showing the processing sequence for generating a sentence from element information according to the conventional method. As shown in Fig. 17, the conventional method uses a conventional sentence generation model M0 that is trained in advance to treat element information as input and sentence as output, and generates sentences from element information without generating a plan that determines the order in which elements corresponding to the element information are written (see, for example, Patent Document 1).

[0096] In a method using a conventional sentence generation model M0, when the amount of element information becomes large and complex, the generated sentence may not be written in the desired order, information may be missing, or the sentence may become redundant. In other words, the order and comprehensiveness of the sentence may be inappropriate. Figure 18 shows an example of a sentence 56Y obtained by inputting a plurality of element information 52X (see Figure 5) related to a medical image 50X (see Figure 4) into the conventional sentence generation model M0. Compared to the sentence 56X shown in Figure 6, the sentence 56Y has an inappropriate order and is difficult for the reader to read.

[0097] 19 shows the results of evaluating sentences generated by the methods according to the first and second embodiments of the present disclosure and sentences generated by a conventional method. Fig. 19 also shows "evaluation scores" calculated by the above-described sentence evaluation model M3, which indicate the appropriateness of the order and degree of coverage of elements in a sentence. As shown in Fig. 19, the sentences generated by the methods according to the first and second embodiments of the present disclosure have higher evaluation scores than sentences generated by the conventional method, indicating that the appropriateness of the order and degree of coverage of elements in a sentence has improved.

[0098] In the above-described embodiments, a plan and a document are generated for a plurality of pieces of element information 52X relating to one region of interest (abnormal shadow N) included in one medical image 50X, but the present invention is not limited to this.

[0099] For example, the acquisition unit 30 may acquire and / or generate a plurality of pieces of element information related to each of a plurality of regions of interest included in one image, and the first generation unit 32 may divide the plurality of pieces of element information into a plurality of groups corresponding to each of the plurality of regions of interest to generate a plan. The second generation unit 34 may generate a sentence in which elements corresponding to the plurality of pieces of element information related to each of the plurality of regions of interest are collectively described.

[0100] 20 shows a plan 54P that assumes a case in which a medical image of the lungs contains abnormal shadows indicating nodules in both the right and left lungs, and a sentence 56P that is generated based on the plan 54P. That is, FIG. 20 assumes a case in which a single medical image containing both the right and left lungs contains multiple abnormal shadows. In plan 54P, element information related to nodules contained in the right lung (group 1) and element information related to nodules contained in the left lung (group 2) are grouped. In this way, the first generation unit 32 may generate a plan by dividing multiple pieces of element information into multiple groups corresponding to the multiple abnormal shadows.

[0101] Alternatively, for example, the acquisition unit 30 may acquire and / or generate a plurality of pieces of element information related to each of the regions of interest included in a plurality of images, and the first generation unit 32 may divide the plurality of pieces of element information into a plurality of groups corresponding to each of the plurality of images to generate a plan. That is, the plurality of pieces of element information acquired and / or generated by the acquisition unit 30 may relate to each of the plurality of images. The second generation unit 34 may generate a sentence in which elements corresponding to the plurality of pieces of element information related to each of the plurality of images are collectively described.

[0102] 21 shows a plan 54Q assuming a case in which a medical image of the lungs and a medical image of the liver of the same subject each contain an abnormal shadow indicating a nodule, and a sentence 56Q generated based on the plan 54Q. That is, FIG. 21 assumes a case in which the medical image of the lungs and the medical image of the liver are different images. In the plan 54Q, element information relating to nodules contained in the lungs (group 1) and element information relating to nodules contained in the liver (group 2) are grouped. In this way, the first generation unit 32 may generate a plan by dividing a plurality of pieces of element information into a plurality of groups corresponding to each of a plurality of images.

[0103] Alternatively, for example, the acquisition unit 30 may acquire and / or generate element information indicating the time points at which images were captured, and the first generation unit 32 may generate a plan in which the order of elements corresponding to related element information is determined based on the time points at which images were captured indicated by the element information. The "time points at which images were captured" may represent, for example, the date and time at which the images were captured, or the imaging phase, such as the arterial phase, portal venous phase, or equilibrium phase, of a contrast agent examination. In this case, the first generation unit 32 may generate a plan by dividing the plurality of element information into a plurality of groups corresponding to the plurality of imaging time points. The second generation unit 34 may generate a sentence in which elements corresponding to the plurality of element information for each of a plurality of images captured at a plurality of different imaging time points are collectively described.

[0104] 22 shows a plan 54R that assumes a case in which abnormal shadows indicating nodules are included in medical images of the lungs of the same subject taken at different times (dates and times), and a sentence 56R that is generated based on the plan 54R. In the plan 54R, element information related to nodules included in medical images taken at a first time point (group 1) and element information related to nodules included in medical images taken at a second time point (group 2) are grouped. In this way, the first generation unit 32 may generate a plan by dividing the element information into multiple groups corresponding to the multiple imaging times.

[0105] 23 shows a plan 54S that assumes that abnormal shadows are included in each medical image of the liver of the same subject in each phase obtained by contrast agent imaging, and a sentence 56S generated based on the plan 54S. In the plan 54S, element information is divided into groups: element information related to the overall characteristics common to each phase (group 1), element information related to the arterial phase (group 2), and element information related to the equilibrium phase (group 3). In this way, the first generation unit 32 may generate a plan by dividing multiple pieces of element information into multiple groups corresponding to multiple imaging phases.

[0106] Furthermore, for example, when generating a plan for element information related to each of a plurality of regions of interest, the first generation unit 32 may derive a severity for each region of interest and generate the plan so that elements corresponding to element information related to a region of interest with a higher severity are described closer to the beginning of the sentence. The severity of a region of interest can be derived, for example, from the size, position, absorption value, etc. of the region of interest. In the example of FIG. 20 , the first generation unit 32 may generate a plan so that elements related to a first region of interest that is relatively large, solid, and accompanied by spicules are described closer to the beginning of the sentence, and elements related to a second region of interest that is relatively small and ground-glass-like are described closer to the end of the sentence.

[0107] Furthermore, for example, when multiple regions of interest can be estimated to be the source and destination of metastasis of tumor cells, the first generation unit 32 may generate a plan such that elements related to the source of metastasis are written at the beginning of a sentence and elements related to the destination of metastasis are written at the end of a sentence. The source and destination of metastasis can be estimated, for example, based on the size, position, and absorption value of the region of interest. Furthermore, when imaging of the same subject is performed at different times (dates and times), estimation can be performed based on the time when an abnormal shadow was detected. In the example of FIG. 21 , the first generation unit 32 may estimate a relatively large lung nodule as the source of metastasis and a relatively small liver tumor as the destination of metastasis, and generate a plan such that elements related to the lung are written at the beginning of a sentence and elements related to the liver are written at the end of a sentence.

[0108] Furthermore, in each of the above-described embodiments, an example has been described in which the first generating unit 32 generates a plan including all of the plurality of pieces of element information acquired and / or generated by the acquiring unit 30, but this is not limiting. When there is a large number of pieces of element information, the first generating unit 32 may generate a plan by selecting and discarding the plurality of pieces of element information acquired and / or generated by the acquiring unit 30 in order to prevent the sentences from becoming too long. In this case, the importance of each piece of element information may be predetermined and stored in the storage unit 22 or the like. For example, with regard to nodules, the importance of element information corresponding to elements that may be evidence of malignancy (e.g., spicules) may be increased, and the importance of element information corresponding to elements that may be evidence of benignity (e.g., clear boundaries) may be decreased.

[0109] Specifically, the first generating unit 32 may generate a plan that specifies not to include in a sentence elements corresponding to element information with a relatively low level of importance among the plurality of element information acquired and / or generated by the acquiring unit 30. For example, when the number of the plurality of element information acquired and / or generated by the acquiring unit 30 exceeds a predetermined threshold, the first generating unit 32 may generate a plan by selecting and discarding the element information in ascending order of importance so as not to include in a sentence the number of element information pieces that exceeds the threshold.

[0110] Furthermore, the first generating unit 32 may generate a plan that specifies that elements corresponding to element information having an importance level lower than a predetermined threshold level among the plurality of element information items acquired and / or generated by the acquiring unit 30 will not be described in the text. Here, the predetermined threshold level may be set arbitrarily by the user. That is, regardless of the number of the plurality of element information items acquired and / or generated by the acquiring unit 30, elements corresponding to element information having an importance level lower than a predetermined threshold level may not be described in the text.

[0111] Furthermore, the first generating unit 32 may generate a plan such that an element corresponding to element information with a higher degree of importance is written closer to the beginning of a sentence.

[0112] The first generation unit 32 may also accept a designation of the degree of conciseness of the sentence generated by the second generation unit 34, and change the number of elements not to be included in the sentence according to the designated degree of conciseness. FIG. 24 shows an example of a screen D2 displayed on the display 24 by the control unit 36 ​​for accepting a designation of the degree of conciseness of the sentence. The screen D2 includes a medical image 50X (see FIG. 4), a plurality of pieces of element information 52X (see FIG. 5) acquired and / or generated by the acquisition unit 30, and a designation field 82 for accepting a designation of the degree of conciseness of the sentence. In this case, the user designates one of the degrees of conciseness of the sentence displayed in the designation field 82 on the screen D2 displayed on the display 24 via the input unit 25. The first generation unit 32 generates a plan based on the plurality of pieces of element information 52X according to the degree of conciseness of the sentence designated by the user.

[0113] For example, it is generally desirable to include in the text an element indicating the presence or absence of calcification in a nodule, since this element can be used as a basis for determining whether the nodule is benign or malignant. On the other hand, some users prefer to omit this element if the nodule is ground-glass, since calcification is generally not observed. Therefore, the first generator 32 may determine whether or not to include in the text an element corresponding to another element based on the presence or absence and degree of certain element information, such as by lowering the importance of element information indicating "calcification" when element information indicating "ground-glass" is present.

[0114] As described above, the information processing device 10 according to each of the above embodiments can generate various plans according to a plurality of different predetermined rules, such as by region (e.g., overall characteristics and peripheral characteristics) for one region of interest, by multiple regions of interest, by multiple images, and by multiple imaging time points. Therefore, the first generation unit 32 may include a plurality of plan generation models for generating various plans according to different rules. For example, the first generation unit 32 may include four plan generation models: a model for generating plans divided into groups by regions for one region of interest, a model for generating plans divided into groups by multiple regions of interest, a model for generating plans divided into groups by multiple images, and a model for generating plans divided into groups by multiple imaging time points.

[0115] Each of the multiple plan generation models is a model such as a CNN or an RNN that has been trained in advance to use element information as input and a plan as output. Each of the multiple plan generation models is trained using, as training data, a set of element information corresponding to elements contained in a previously generated sentence and a plan in which the order in which the elements are written in the sentence is determined, but the training data differs depending on the model. For example, a model for generating plans divided into groups by region related to a single region of interest uses, as training data, plans divided into groups by region related to a single region of interest. On the other hand, a model for generating plans divided into groups by multiple regions of interest uses, as training data, plans divided into groups by multiple regions of interest.

[0116] The first generating unit 32 may generate a plan by selectively using at least one of the plurality of plan generation models. Regarding which plan generation model to select, the first generating unit 32 may select an optimal plan generation model based on, for example, the plurality of element information acquired and / or generated by the acquiring unit 30.

[0117] Furthermore, for example, the first generating unit 32 may accept the designation of at least one rule from among a plurality of different predetermined rules, and may divide a plurality of pieces of element information into groups according to the designated rule. Specifically, the first generating unit 32 may accept a user's designation as to which plan generation model to select from a plurality of plan generation models.

[0118] 25 shows an example of a screen D3 for accepting the designation of a plurality of different rules, which is displayed on the display 24 by the control unit 36. The screen D3 includes a medical image 50X (see FIG. 4), a plurality of pieces of element information 52X (see FIG. 5) acquired and / or generated by the acquisition unit 30, and a designation field 80 for accepting the designation of a plurality of different rules. In this case, the user designates, via the input unit 25, at least one of the plurality of different rules displayed in the designation field 80 on the screen D3 displayed on the display 24.

[0119] The first generating unit 32 divides the plurality of pieces of element information 52X into groups according to at least one rule specified by the user. Specifically, the first generating unit 32 generates a plan by selectively using, from among the plurality of plan generation models, a plan generation model that has been trained in advance using training data that complies with at least one rule specified by the user.

[0120] Furthermore, in the above-described embodiments, medical images are used as an example of images, but the technology of the present disclosure can also be used with images other than medical images. For example, the technology of the present disclosure can be applied when creating a report on images (e.g., CT images, visible light images, infrared images, etc.) taken during non-destructive testing of civil engineering structures, industrial products, piping, etc. In this case, the element information related to the image can be information indicating at least one of the name, characteristics, measurement values, and position of a region of interest contained in the image, as well as the imaging method, imaging conditions, and imaging date and time related to the image capture.

[0121] Furthermore, in the above embodiment, the following various processors can be used as the hardware structure of processing units that perform various processes, such as the acquisition unit 30, the first generation unit 32, the second generation unit 34, the control unit 36, and the evaluation unit 38. As described above, the various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD), a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0122] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0123] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by computers such as client and server. Second, a form in which a processor is used to realize the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by systems on chips (SoCs). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0124] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0125] In the above embodiment, the information processing program 27 is pre-stored (installed) in the storage unit 22, but this is not limiting. The information processing program 27 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 27 may also be downloaded from an external device via a network. Furthermore, the technology disclosed herein extends to not only information processing programs but also storage media that non-temporarily store information processing programs.

[0126] The technology of the present disclosure can also be achieved by appropriately combining the above-described exemplary embodiments. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or new elements may be substituted from the above-described description and illustrations, within the scope of the gist of the technology of the present disclosure. [Explanation of symbols]

[0127] 1. Information Processing Systems 2. Imaging equipment 3 Image Reading Workshop 4. Medical Workshop 5 Image Server 6. Image Database 7 Report Server 8 Report DB 9 Network 10. Information processing equipment 21 CPU 22 Memory section 23 Memory 24 displays 25 Input section 26 Network I / F 27 Information Processing Program 28 Bus 30 Acquisition Department 32 1st generation part 34 Second generation part 36 Control Unit 38 Evaluation Department 50X Medical Image 52X Element Information 54P~54S, 54X Plan 56P~56S, 56X, 56Y text 64A~64C Plan Candidates 66A~66C Sentence candidates 68A~68C rating score 80, 82 Specified field D1~D3 screen M1 Plan Generation Model M2 sentence generation model M3 Text Evaluation Model N Abnormal shadow

Claims

1. at least one processor; The processor: Obtaining multiple element information about an image; generating a plan including at least one of the plurality of pieces of element information arranged in accordance with the order in which the elements corresponding to the plurality of pieces of element information are described in a sentence in which the elements are described, by inputting the plurality of pieces of element information into a first trained model that has been trained in advance to have the plurality of pieces of element information as input and the plan as output, and receiving the plan from the first trained model; The sentence in which elements corresponding to the element information included in the plan are written in the order of description is generated by inputting the plan into a second trained model that has been trained in advance to take the plan as an input and the sentence as an output, and receiving the sentence from the second trained model. Information processing device.

2. The processor: Dividing the plurality of pieces of element information into groups; The plan is generated with the order of description determined for each group. The information processing device according to claim 1 .

3. The processor: Accepting designation of at least one rule from among a plurality of different rules that are predetermined for dividing the plurality of pieces of element information into the groups; Dividing the plurality of pieces of element information into groups according to the specified rules The information processing device according to claim 2 .

4. The processor: acquiring a plurality of pieces of element information relating to each of a plurality of regions of interest included in the image; The plurality of pieces of element information are divided into a plurality of groups corresponding to the plurality of regions of interest, respectively.

4. The information processing device according to claim 2 or 3.

5. the plurality of pieces of element information relate to each of a plurality of images, The processor: Generate a sentence in which elements corresponding to the plurality of element information related to each of the plurality of images are collectively described. The information processing device according to any one of claims 1 to 4.

6. The processor: acquiring element information indicating the time of capturing the image; The plan is generated based on the shooting time indicated by the element information, in which the description order of elements corresponding to related element information is determined. The information processing device according to any one of claims 1 to 3.

7. an importance level is determined in advance for each of the element information; The processor: The plan is generated in which an element corresponding to the element information whose importance is lower than a predetermined threshold among the plurality of element information is not to be described in the sentence. The information processing device according to any one of claims 1 to 6.

8. an importance level is determined in advance for each of the element information; The processor: The plan is generated in which an element corresponding to element information having a relatively low degree of importance among the plurality of element information is not to be described in the sentence. The information processing device according to any one of claims 1 to 7.

9. The processor: Accepting a designation of the degree of conciseness of the sentence; The number of elements not described in the sentence is changed according to the degree of conciseness.

9. The information processing device according to claim 7 or 8.

10. The first trained model is trained using, as training data, a set of element information corresponding to the elements included in the sentence generated in the past and a plan in which the order in which the elements are written in the sentence is determined. The information processing device according to claim 1 .

11. The processor: generating a plurality of different plan candidates for the plurality of element information; generating the sentence for each of the plan candidates; Evaluate each of the sentences, Based on the results of the evaluation, the candidate plans to be adopted are determined. The information processing device according to any one of claims 1 to 10.

12. The processor: The evaluation is performed based on at least one of the order of description and the degree of coverage of the elements included in the generated sentence. The information processing device according to claim 11.

13. The processor: generating a plurality of different plan candidates for the plurality of element information; Accepting designation of the candidate plan to be adopted from among the plurality of different candidate plans. The information processing device according to any one of claims 1 to 10.

14. The processor: acquiring the image; The element information is generated based on the acquired image. The information processing device according to any one of claims 1 to 13.

15. further comprising an input unit, The processor: The element information is generated based on the information input via the input unit. The information processing device according to any one of claims 1 to 14.

16. The element information is information indicating at least one of the name, characteristics, measurement values, and position of a region of interest included in the image, and the imaging method, imaging conditions, and imaging date and time of the image. The information processing device according to any one of claims 1 to 15.

17. the image is a medical image, The element information is information indicating at least one of the name, characteristics, position, and presumed disease name of a region of interest included in the medical image, and the imaging method, imaging conditions, and imaging date and time of imaging of the medical image, The region of interest is at least one of a region of a structure included in the medical image and a region of an abnormal shadow included in the medical image. The information processing device according to any one of claims 1 to 16.

18. Obtaining multiple element information about an image; generating a plan including at least one of the plurality of pieces of element information arranged in accordance with the order in which the elements corresponding to the plurality of pieces of element information are described in a sentence in which the elements are described, by inputting the plurality of pieces of element information into a first trained model that has been trained in advance to have the plurality of pieces of element information as input and the plan as output, and receiving the plan from the first trained model; The sentence in which elements corresponding to the element information included in the plan are written in the order of description is generated by inputting the plan into a second trained model that has been trained in advance to take the plan as an input and the sentence as an output, and receiving the sentence from the second trained model. An information processing method in which processing is performed by a computer.

19. Obtaining multiple element information about an image; generating a plan including at least one of the plurality of pieces of element information arranged in accordance with the order in which the elements corresponding to the plurality of pieces of element information are described in a sentence in which the elements are described, by inputting the plurality of pieces of element information into a first trained model that has been trained in advance to have the plurality of pieces of element information as input and the plan as output, and receiving the plan from the first trained model; The sentence in which elements corresponding to the element information included in the plan are written in the order of description is generated by inputting the plan into a second trained model that has been trained in advance to take the plan as an input and the sentence as an output, and receiving the sentence from the second trained model. An information processing program that causes a computer to execute a process.

Images