Information processing device, method, and program

Abstract

In the present invention, a processor uses a trained neural network to derive a characteristic score for each of a plurality of characteristic items determined in advance in relation to a structure of interest contained in an image. The processor corrects a characteristic score for at least one of the plurality of characteristic items by referring to information representing a relationship between the plurality of characteristic items, said information being derived by analysing a co-occurrence relationship of descriptions of characteristics contained in a plurality of sentences. On the basis of the corrected characteristic score, the processor derives a determination result for the plurality of characteristic items in relation to the structure of interest.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, method, and program.

Background Art

[0002] Medical images are analyzed by CAD (Computer-Aided Diagnosis) using a learned model constructed by learning a neural network by deep learning or the like, and the shape, density, position, size, and other properties of a structure of interest such as a lesion included in the medical image are discriminated. vinegar This is being done. From the learned model, scores representing the salience for each of a plurality of property items are output, and by comparing the output scores with a threshold value, the properties of the structure of interest are discriminated.

[0003] On the other hand, when using a learned model, if the learning is not sufficient, a discrimination result that is an impossible combination regarding properties may be obtained. For example, regarding a lung lesion, when it is discriminated as positive for the property of "smooth edge", it may be discriminated as positive for "spicule" that includes a linear structure at the periphery. For this reason, a method for improving the discrimination accuracy of properties has been proposed. For example, in "Holistic and Comprehensive Annotation of Clinically Significant Findings on Diverse CT Images: Learning from Radiology Reports and Label Ontology, Ke Ya et al., 1904.04661v2 [cs.CV] 27 Apr 2019", a method using a matrix representing the relationship of discriminated properties has been proposed. According to the method described in the literature by Ke Ya et al., by correcting the property scores for each property output by the learned model with a matrix representing the relationship, it is possible to prevent a discrimination result that is an impossible combination of properties from being output.

Summary of the Invention

[0004] However, in the method described by Ke Ya et al., the matrix representing the relationships between properties (hereinafter referred to as the relationship matrix) is constructed simultaneously with the training of the neural network. Therefore, in order to train both the neural network and the relationship matrix so that a trained model that can accurately discriminate properties can be constructed, a large amount of training data relating structures of interest and properties contained in medical images is required. However, there is a limit to the number of medical images necessary to cover all of multiple properties.

[0005] This disclosure is made in view of the above circumstances and aims to accurately determine the properties of structures of interest contained in medical images without using a large amount of training data. [Means for solving the problem]

[0006] The information processing device according to this disclosure comprises at least one processor, The processor uses a trained neural network to derive characteristic scores for each of several predetermined characteristic items regarding the structure of interest contained in the image. By analyzing the co-occurrence relationships of descriptions of properties in multiple texts, and referring to information representing the relationships between multiple property items, the property score for at least one of the multiple property items is modified. Based on the modified trait scores, discriminant results for multiple trait items regarding the structure of interest Derive the result.

[0007] Furthermore, in the information processing device described herein, the processor further trains a neural network using training data in which structures of interest and multiple characteristic items of the structures of interest contained in medical images have been identified. It may also be possible to update the information representing relationships based on the learning results.

[0008] Furthermore, in the information processing device according to this disclosure, the information representing relationships may be a relationship matrix in which elements are defined with weights that increase as the co-occurrence relationship between multiple property items becomes stronger.

[0009] Furthermore, in the information processing device described herein, the weights may be scaled to a predetermined range.

[0010] Furthermore, in the information processing apparatus described herein, the processor presents a relationship matrix, The relationship matrix may also be modified by accepting adjustments to the weights in the presented relationship matrix.

[0011] Furthermore, in the information processing device described herein, the learned neural network is constructed by machine learning a convolutional neural network. The processor may modify the trait score using a single fully connected layer to which relationship-representing information is applied, and which receives the output of the convolutional neural network as input.

[0012] The information processing method disclosed herein uses a trained neural network to derive characteristic scores for each of several predetermined characteristic items regarding the structure of interest contained in an image. By analyzing the co-occurrence relationships of descriptions of properties in multiple texts, and referring to information representing the relationships between multiple property items, the property score for at least one of the multiple property items is modified. Based on the modified characteristic scores, the discrimination results for multiple characteristic items regarding the structure of interest are derived.

[0013] Furthermore, the information processing method described herein may be provided as a program for causing a computer to execute it. [Effects of the Invention]

[0014] According to the present disclosure, even without using a large amount of teacher data, the properties of the structure of interest included in the medical image can be accurately determined.

Brief Description of the Drawings

[0015] [Figure 1] The figure showing the schematic configuration of the medical information system to which the information processing apparatus according to the embodiment of the present disclosure is applied [Figure 2] The figure showing the schematic configuration of the information processing apparatus according to the present embodiment [Figure 3] The functional configuration diagram of the information processing apparatus according to the present embodiment [Figure 4] The figure schematically showing the neural network constituting the discrimination model [Figure 5] The figure showing the teacher data [Figure 6] The figure for explaining the derivation of the relational matrix [Figure 7] The figure showing the relational matrix [Figure 8] The figure showing the creation screen of the radiology report [Figure 9] The flowchart showing the processing performed in the present embodiment [Figure 10] The figure showing the word vector in the feature space [Figure 11] The figure showing the correction screen of the relational matrix

Modes for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. First, the configuration of the medical information system 1 to which the information processing apparatus according to the present embodiment is applied will be described. FIG. 1 is a diagram showing the schematic configuration of the medical information system 1. The medical information system 1 shown in FIG. 1 is a system for performing imaging of the imaging target site of the subject, storage of the medical image obtained by the imaging, reading of the medical image by the radiologist and creation of the radiology report, and viewing of the radiology report by the doctor of the requesting department and detailed observation of the medical image to be read, based on the examination order from the doctor of the department using a known ordering system.

[0017] Each device is a computer on which an application program is installed to function as a component of the medical information system 1. The application program is stored in a storage device of a server computer connected to network 10, or in network storage, in a state that is accessible from the outside, and is downloaded and installed on the computer upon request. Alternatively, it is recorded on a recording medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) and distributed, and It is installed on the computer from the storage medium.

[0018] The imaging device 2 is a device (modality) that generates a medical image representing the area to be diagnosed by imaging the area of ​​the subject to be diagnosed. Specifically, this includes a plain X-ray imaging device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, and a PET scanner. (Positron Emission Tomography) equipment, etc. Medical images generated by imaging device 2. The image is sent to image server 5 and stored in image database 6.

[0019] The Image Interpretation WS3 is a computer used, for example, by a radiologist in the radiology department for interpreting medical images and creating interpretation reports, and it incorporates the information processing device 20 according to this embodiment. The Image Interpretation WS3 performs tasks such as requesting the image server 5 to view medical images, performing various image processing on medical images received from the image server 5, displaying medical images, and accepting input of findings text related to medical images. The Image Interpretation WS3 also performs analysis processing on medical images and input findings text, assists in creating interpretation reports based on the analysis results, requests the report server 7 to register and view interpretation reports, and displays interpretation reports received from the report server 7. These processes are performed by the Image Interpretation WS3 executing software programs for each process.

[0020] The Clinical WS4 is a computer used by physicians in various departments for detailed examination of images, viewing of image interpretation reports, and creation of electronic medical records. It consists of a processing unit, display devices such as a display, and input devices such as a keyboard and mouse. The Clinical WS4 handles requests to view images from the Image Server 5, displays images received from the Image Server 5, requests to view image interpretation reports from the Report Server 7, and displays image interpretation reports received from the Report Server 7. These processes are carried out by the Clinical WS4 executing software programs for each process.

[0021] Image Server 5 is a general-purpose computer with software programs installed that provide the functionality of a Database Management System (DBMS). Image Server 5 also has storage that houses the image DB 6. This storage may be a hard disk drive connected to Image Server 5 via a data bus, or it may be a Network Attached Storage (NAS) connected to Network 10. It may also be a disk device connected to a SAN (Storage Area Network). Furthermore, when the image server 5 receives a request to register a medical image from the imaging device 2, it formats the medical image into a database format and registers it in the image DB 6.

[0022] Image DB6 stores image data and associated information of medical images acquired by imaging device 2. The associated information includes, for example, an image ID (identification) to identify individual medical images, a patient ID to identify the subject, an examination ID to identify the examination, a unique ID (UID: unique identification) assigned to each medical image, and information about the generation of the medical image. This includes information such as the date and time of the examination, the type of imaging device used to acquire the medical image, patient information such as the patient's name, age, and gender, the examination site (imaging site), imaging information (imaging protocol, imaging sequence, imaging method, imaging conditions, use of contrast agent, etc.), and, if multiple medical images were acquired in a single examination, the series number or acquisition number.

[0023] Furthermore, when the image server 5 receives a viewing request from the image interpretation WS3 and the medical treatment WS4 via the network 10, it searches for medical images registered in the image database 6 and sends the retrieved medical images to the requesting image interpretation WS3 and medical treatment WS4.

[0024] The report server 7 incorporates software programs that provide the functionality of a database management system to a general-purpose computer. When the report server 7 receives a request to register an image interpretation report from the image interpretation WS3, it formats the image interpretation report into a database format and registers it in the report DB8.

[0025] The report DB8 registers image interpretation reports that include at least the findings statement created in the image interpretation WS3. The image interpretation report may include, for example, the medical image being interpreted, an image ID to identify the medical image, a radiologist ID to identify the radiologist who performed the interpretation, the name of the lesion, location information of the lesion, information for accessing the medical image containing a specific region, and characteristic information.

[0026] Furthermore, when the report server 7 receives a request to view or send an image interpretation report from the image interpretation WS3 and medical treatment WS4 via the network 10, it searches the report DB8 for the image interpretation report registered therein and sends the retrieved report to the requesting image interpretation WS3 and medical treatment WS4.

[0027] In this embodiment, the medical image is a three-dimensional CT image consisting of multiple tomographic images of the lung as the target of diagnosis. By interpreting the CT image, an interpretation report is created that includes findings regarding structures of interest, such as abnormal shadows, contained in the lung. The medical image is not limited to CT images; any medical image such as MRI images and simple two-dimensional images obtained by a simple X-ray imaging device can be used.

[0028] Network 10 is a wired or wireless local area network that connects various devices within the hospital. If the image interpretation WS3 is installed in another hospital or clinic, Network 10 may be configured to connect the local area networks of each hospital via the Internet or a dedicated line.

[0029] Next, the information processing device according to this embodiment will be described. Figure 2 illustrates the hardware configuration of the information processing device according to this embodiment. As shown in Figure 2, the information processing device 20 includes a CPU (Central Processing Unit) 11, non-volatile storage 13, and temporary storage. It includes memory 16 as a region. The information processing device 20 also includes a display 14 such as an LCD display, input devices 15 such as a keyboard and mouse, and a network I / F (Interface) 17 connected to the network 10. CPU 11, storage 13, The display 14, input device 15, memory 16, and network I / F 17 are connected to the bus 18. The CPU 11 is an example of a processor in this disclosure.

[0030] Storage 13 is implemented by HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. Information processing programs are stored in storage 13 as a storage medium. The CPU 11 reads the information processing program 12 from storage 13, expands it into memory 16, and executes the expanded information processing program 12.

[0031] Next, the functional configuration of the information processing device according to this embodiment will be described. Figure 3 is a diagram showing the functional configuration of the information processing device according to this embodiment. As shown in Figure 3, the information processing device 20 includes an information acquisition unit 21, an image analysis unit 22, a learning unit 23, a relationship derivation unit 24, and a display control unit 25. The CPU 11 executes the information processing program 12, and the CPU 11 functions as the information acquisition unit 21, the image analysis unit 22, the learning unit 23, the relationship derivation unit 24, and the display control unit 25.

[0032] The information acquisition unit 21 acquires medical images from the image server 5 to create an image interpretation report based on instructions from the radiologist operator via the input device 15. The information acquisition unit 21 also acquires the image interpretation report from the report server 7.

[0033] The image analysis unit 22 analyzes medical images and derives discrimination results for multiple characteristic items regarding structures of interest contained in the medical images. To this end, the image analysis unit 22 has an extraction model 22A that extracts abnormal shadows in medical images as structures of interest, and a discrimination model 22B that derives discrimination results for each of a predetermined set of characteristic items regarding the extracted structures of interest.

[0034] The extraction model 22A consists of a neural network that has been machine-learned using deep learning or the like with training data to extract structures of interest from medical images. For example, well-known neural networks such as convolutional neural networks (CNNs) and support vector machines (SVMs) can be used as the extraction model 22A. Furthermore, the extraction model 22A may extract structures of interest from medical images using template matching or the like.

[0035] The discrimination model 22B also consists of a neural network that has undergone machine learning using deep learning or the like with training data. This embodiment has a distinctive feature in the discrimination model 22B, so the discrimination model 22B will be described below.

[0036] Figure 4 is a schematic diagram showing the neural network that constitutes the discriminant model. As shown in Figure 4, the discriminant model 22B has a convolutional neural network (hereinafter referred to as CNN) 31, a fully connected layer 32, and an output layer 33. The CNN 31 has an input layer 34, multiple convolutional layers 35, multiple pooling layers 36, and a fully connected layer 37. The multiple convolutional layers 35 and multiple pooling layers 36 are arranged alternately between the input layer 34 and the fully connected layer 37. Note that the configuration of the CNN 31 is not limited to the example in Figure 4. For example, the CNN 31 may have one convolutional layer 35 and one pooling layer 36 between the input layer 34 and the fully connected layer 37.

[0037] When a medical image is input to the image analysis unit 22, the extraction model 22A extracts structures of interest contained in the medical image. At this time, the size and location of the structures of interest are also extracted. The image analysis unit 22 then uses the CNN 31 of the discrimination model 22B to derive property scores for multiple property items regarding the structures of interest, and the fully connected layer 32 processes the property scores as described later. The output layer 33 corrects the output and outputs the discrimination results for each property item.

[0038] The learning unit 23 trains the CNN 31 in the discriminant model 22B. Figure 5 shows an example of training data for training the CNN in the discriminant model. As shown in Figure 5, the training data 40 includes an interest structure image 42 extracted from a medical image 41 and a discrimination result 43 for several characteristic items that have been pre-derived regarding the interest structure. In Figure 5, a medical image 41 is shown for illustrative purposes, but only the interest structure image 42 is used as training data 40. Therefore, in reality, the training data 40 consists of the interest structure image 42 and the characteristic discrimination result 43. The discrimination result 43 becomes the ground truth data.

[0039] In this embodiment, the abnormal shadow is a pulmonary nodule, and the discrimination result 43 is the discrimination result for multiple characteristic items of the pulmonary nodule. The characteristic items are the type of absorption value (solid, partially solid, and ground glass), presence or absence of spicules, and bronchial transparency. image The presence or absence of a ridge, the presence or absence of a smooth margin, the presence or absence of lobulation, and the presence or absence of pleural invagination are used. However, the examples of characteristic items are not limited to these.

[0040] The characteristics of the structure of interest image 42 shown in Figure 5, as determined by the attenuation value, are as follows: solid is absent, partially solid is present, ground-glass opacity is absent, spicules are absent, bronchial radiolucency is present, smooth margins are absent, lobulated is absent, and pleural invagination is absent. In Figure 5, a + is assigned to present and a - to absent. The discrimination model 22B is constructed by training a CNN 31 using a large amount of training data as shown in Figure 5.

[0041] The learning unit 23 inputs the structure of interest image 42 to the input layer 34 of the CNN 31. As a result, multiple convolutional layers 35 and pooling layers 36 derive features from the structure of interest image 42, and the fully connected layer 37 derives a characteristic score for each of the multiple characteristic items. The characteristic score is a score that indicates the prominence of the characteristic for each characteristic item. The characteristic score takes a value between 0 and 1, for example, and the larger the characteristic score, the more prominent the characteristic for that characteristic item is.

[0042] The learning unit 23 then derives the difference between the characteristic score for each of the multiple characteristic items output by the CNN 31 and the respective scores for the characteristic items included in the characteristic classification results 43 of the training data 40 as the loss. Here, the scores for each characteristic item included in the classification results 43 of the training data 40 are: solid = 0.0, partially solid = 1.0, ground-glass = 0.0, spicule = 0.0, bronchial radiolucency = 1.0, marginal smooth = 0.0, lobulated = 0.0, and pleural invagination = 0.0. The learning unit 23 then derives the difference in scores between corresponding characteristic items as the loss. The learning unit 23 then adjusts parameters such as the kernel coefficients in the convolutional layer 35, the connection weights between each layer, and the connection weights in the fully connected layer 37 to minimize the loss.

[0043] For example, backpropagation can be used to adjust the parameters. The learning unit 23 repeatedly adjusts the parameters until the loss falls below a predetermined threshold. As a result, when an image of a structure of interest is input, the parameters are adjusted to output characteristic scores for multiple characteristic items, and a trained CNN 31 is constructed.

[0044] On the other hand, the fully connected layer 32 of the discrimination model 22B modifies the discrimination result output by the CNN 31 using information representing the relationships between multiple characteristic items. The information representing the relationships is a matrix in which a larger weight is defined for each stronger co-occurrence relationship between multiple characteristic items. The relationship derivation unit 24 derives this matrix.

[0045] Here, co-occurrence, in the field of natural language processing, refers to the simultaneous appearance of two strings in any given document or sentence. For example, regarding the word "spicula"... The term "spicula" is often used in conjunction with "partially enriched" in medical reports, but it is rarely used in conjunction with "frosted glass." For this reason, "spicula" and "partially enriched" co-occur more frequently, while "spicula" and "frosted glass" rarely co-occur.

[0046] In this embodiment, the relationship derivation unit 24 analyzes a large number of observation sentences described in the image interpretation report to derive the co-occurrence relationships of words contained in the observation sentences, and derives a relationship matrix in which a larger weight is defined for each greater co-occurrence relationship. Figure 6 is a diagram illustrating the derivation of the relationship matrix. The image interpretation report used when deriving the relationship matrix is ​​obtained from the report server 7 by the information acquisition unit 21.

[0047] First, the relationship derivation unit 24 derives structured labels by structuring the expressions contained in the findings sentences included in the image interpretation report. To this end, the relationship derivation unit 24 first selects findings sentences describing lung nodules from a large number of image interpretation reports. Then, the relationship derivation unit 24 derives named expressions related to the lesion from the selected findings sentences. Named expressions represent the characteristics of the lesion contained in the findings sentences. For example, from the findings sentence 44 shown in Figure 6, "A 13 mm partially solid nodule is observed in the right lung S8. Spicula are observed at the periphery, and bronchial radiolucency is observed inside," the unit derives "partially solid," "spicule," and "bronchial radiolucency" as named expressions.

[0048] Furthermore, the relationship derivation unit 24 determines the factual validity of the derived named expressions. Specifically, the relationship derivation unit 24 determines whether the named expressions represent a negative result, a positive result, or a suspected result, and derives a determination result. For example, in this embodiment, the named expressions in the findings statement 44 are "partially solid," "spicules," and "bronchial radiolucency." The findings statement 44 states, "A partially solid nodule is observed," and "Spicules are observed on the periphery, and bronchial radiolucency is observed internally." Therefore, the relationship derivation unit 24 determines that the factual validity of "partially solid," "spicules," and "bronchial radiolucency" is positive. The relationship derivation unit 24 then derives structured labels with factual validity added to the named expressions.

[0049] In other words, as shown in Figure 6, the relationship derivation unit 24 derives structured labels 45 of "partially solidified +", "spicules +", and "bronchial radiolucency +" from the findings statement 44, assigning a + sign to indicate a positive result. If the findings statement is negative, such as "~ not observed," a - sign should be assigned, and if there is suspicion, a ± sign should be assigned.

[0050] Similarly, the relationship derivation unit 24 derives structured labels 47 of "solid+", "lobulated+", and "pleural invagination+" from the findings statement 46, "An 8 mm solid nodule is observed in left lung S3. It is lobulated and shows pleural invagination."

[0051] The relationship derivation unit 24 then counts the number of positive descriptions for the same characteristic item in the same finding statement for the nodule characteristics: "solid," "partially solid," "ground-glass," "spicule," "bronchial radiolucency," "smooth margin," "lobulated," and "pleural invagination." The counted number represents the degree of co-occurrence relationship for the characteristic item. Therefore, the relationship derivation unit 24 determines the weights between the characteristic items according to the counted number and derives a relationship matrix with the weights as elements.

[0052] For example, suppose the analysis reveals that there are 8,000 findings that simultaneously describe "partially solid" and "bronchial radiolucency," and 4 findings that simultaneously describe "partially solid" and "ground-glass opacity." In this case, a larger weight is assigned to the combination of "partially solid" and "bronchial radiolucency," and a smaller weight is assigned to the combination of "partially solid" and "ground-glass opacity."

[0053] Figure 7 shows an example of a relationship matrix. The relationship matrix 48 shown in Figure 7 has characteristic items assigned to the rows and columns for illustrative purposes, but in reality, it is an 8x8 matrix with only weights as elements. It is preferable to scale the weights so that they fall within a predetermined range. For example, it is preferable to scale the weights so that they are between -0.2 and +0.2.

[0054] As shown in the relationship matrix 48 in Figure 7, for example, for the partially solid characteristic item, the weight for the bronchial radiolucency characteristic item is a large 0.20, while the weight for the solid and ground-glass characteristic items is a small -0.13. This indicates that when the characteristic of a structure of interest included in the image is partially solid, a bronchial radiolucency often appears simultaneously, and the characteristic is rarely solid or ground-glass. In relation matrix 48, the weight between identical characteristic items is 1.0.

[0055] The relationship matrix 48 derived by the relationship derivation unit 24 is applied to the fully connected layer 32 of the discrimination model 22B. When the image analysis unit 22 discriminates the properties of structures of interest contained in the input medical image, the fully connected layer 32 modifies the property scores for each property item output by the trained CNN 31 using the relationship matrix 48.

[0056] In this embodiment, the CNN31 derives property scores for the property items "solid," "partially solid," "ground-glass," "spicules," "bronchial radiolucency," "smooth margins," "lobulated," and "pleural invagination." The property scores for the property items "solid," "partially solid," "ground-glass," "spicules," "bronchial radiolucency," "smooth margins," "lobulated," and "pleural invagination" are denoted as a1 to a8. The modified property scores are denoted as b1 to b8. If the vector with the original property scores a1 to a8 as elements is Va, the vector with the modified property scores b1 to b8 as elements is Vb, and the relationship matrix 48 is M, then in the fully connected layer 32, the calculation Vb = M·Va is performed.

[0057] For example, the corrected characteristic score b1 for "fullness" output by CNN31 is derived by the calculation b1 = a1 × 1.0 + a2 × (-0.13) + a3 × (-0.2) + a4 × 0.0 + a5 × 0.01 + a6 × 0.01 + a7 × 0.05 + a8 × 0.03. The discriminant model 22B outputs the characteristic score corrected by the fully connected layer 32 as the characteristic score for the input medical image.

[0058] Figure 4 shows the state after the property scores output by CNN31, which were solid=0.8, partially solid=0.5, ground-glass=0.1, and pleural invagination=0.1, have been corrected by the fully connected layer 32 to solid=0.7, partially solid=0.3, ground-glass=0.1, and pleural invagination=0.1.

[0059] The display control unit 25 displays a screen for creating a medical image interpretation report. Figure 8 shows the screen for creating a medical image interpretation report. As shown in Figure 8, the creation screen 50 has an image display area 51, a properties display area 52, and a text display area 53. The medical image G0 for which the medical image interpretation report is to be created is displayed in the image display area 51. A rectangular mark 55 is assigned to the structures of interest 54 contained in the medical image G0.

[0060] The characteristics display area 52 displays the results of the discrimination of multiple characteristics items determined by the image analysis unit 22 for the medical image G0. In Figure 8, the characteristics display area 52 displays the discrimination results for the following characteristics items: "solid +", "partially solid -", "ground-glass -", ​​"spicul -", "bronchial radiolucency -", "smooth margin +", "lobulated -", and "pleural invagination -". In addition to these discrimination results, the location and size of the abnormal shadow ("Location of abnormal shadow: left upper lung S1+2" and "Size of abnormal shadow: 24 mm") are also displayed.

[0061] The text display area 53 displays the findings entered by the radiologist based on the medical image G0 and the results of the characteristic assessment.

[0062] Additionally, a confirmation button 56 is displayed below the image display area 51. The radiologist enters their findings in the text display area 53 and then selects the confirmation button 56. This transfers the medical text displayed in the text display area 53 to the radiology report. The radiology report, with the findings added, is then sent to the report server 7 along with the medical image G0 for storage.

[0063] Next, the processing performed in this embodiment will be described. Figure 9 is a flowchart showing the processing performed in this embodiment. The medical image to be interpreted is assumed to be acquired from the image server 5 by the information acquisition unit 21 and stored in the storage 13. Processing starts when the radiologist gives instructions to create an interpretation report, and the image analysis unit 22 analyzes the medical image to determine the properties of the structure of interest contained in the medical image G0. Specifically, the extraction model 22A of the image analysis unit 22 extracts the structure of interest from the medical image G0 (step ST1), and the CNN 31 of the discrimination model 22B derives a property score for each of the multiple property items with respect to the structure of interest (step ST2). Then, the property score is modified by the relationship matrix 48 applied to the fully connected layer 32 (step ST3), and the discrimination results for the multiple property items are output from the output layer 33 based on the modified property score (step ST4).

[0064] Then, the display control unit 25 displays the image interpretation report creation screen on the display 14 (step ST5), and the process ends. The radiologist can create the image interpretation report on the image interpretation report creation screen as described above. The created image interpretation report is sent to the report server 7 and saved as described above.

[0065] Thus, in this embodiment, the property score for each of the multiple property items is modified by referring to the relationship matrix 48, and the discrimination results for the multiple property items with respect to the structure of interest are derived based on the modified property scores. By creating the relationship matrix 48 in advance in this way, it is possible to prevent the deriving of discrimination results for impossible combinations of property items without using a large amount of training data. Therefore, according to this embodiment, the properties of the structure of interest contained in medical images can be accurately determined without using a large amount of training data.

[0066] In the above embodiment, the learning unit 23 may further train the discriminant model 22B using training data to update the relationship matrix 48. In this case, although the above embodiment derives the loss as the difference between the characteristic score for each of the multiple characteristic items output by the CNN 31 and the respective scores of the characteristic items included in the discriminant result 43 of the training data 40, instead, the loss may be derived as the difference between the corrected characteristic score output by the fully connected layer 32 and the respective scores of the characteristic items included in the discriminant result 43 of the training data 40.

[0067] The learning unit 23 then adjusts parameters such as the kernel coefficients in the convolutional layer 35, the connection weights between each layer, and the connection weights in the fully connected layer 37 in order to reduce the derived loss. Furthermore, the learning unit 23 updates the relationship matrix 48 by adjusting the values ​​of each element of the relationship matrix 48.

[0068] In this way, by further training the discrimination model 22B using training data and updating the relationship matrix 48, the image analysis unit 22 can more accurately discriminate the properties of structures of interest contained in medical images.

[0069] In the above embodiment, a relationship matrix 48 is derived based on the co-occurrence relationships between multiple characteristic items by analyzing a large number of image interpretation reports, but the invention is not limited to this. For example, the word2ve method can be used to vectorize the words contained in a large number of image interpretation reports. Alternatively, the weights of the relationship matrix may be defined based on the distance between vectorized words in the feature space. The word2vec method is described in detail in "Efficient Estimation of Word Representations in Vector Space, Tomas Mikolov et al., 1301.3781v3, 7 Sep 2013". It is stated.

[0070] Using the word2vec method, the words included in a large number of radiology reports are vectorized, and a state where the words are plotted in the feature space is schematically shown in FIG. 10. Note that although the dimension of the vectorized words (hereinafter referred to as word vectors) is multi-dimensional, in FIG. 10, the word vectors are shown in two dimensions. Further, FIG. 10 shows a state where word vectors V1, V2, and V3 for three words, "partial filling", "bronchial translucency image", and "ground glass", are plotted in the feature space. As shown in FIG. 10, in the feature space, the distance between the word vector V1 of "partial filling" and the word vector V2 of "bronchial translucency image" is close, but the distance between the word vector V1 of "partial filling" and the word vector V3 of "ground glass" is far.

[0071] Therefore, in the relationship derivation unit 24, word vectors are derived for the words included in the radiology report, and in the feature space of the word vectors, the distance d between the word vectors is derived. As the distance d, any distance such as the Euclidean distance and the Mahalanobis distance can be used. Then, a threshold value where α < β is set, and when the distance d < α, the weight is +0.20, when the distance d > β, the weight is -0.20, and when α < d < β, the weight can be derived by the operation of 0.4×(d - α) / (β - α) - 0.2.

[0072] Note that instead of the weight according to the above distance d, the cosine similarity (inner product) between the word vectors may be used as the weight. In this case, since the cosine similarity is a value between -1.0 and +1.0, the derived cosine similarity value may be directly defined in the relationship matrix. Also, the cosine similarity may be scaled to a value between -0.20 and +0.20 to define the weight.

[0073] Further, in the above embodiment, the relationship derivation unit 24 derives the relationship matrix by analyzing the radiology report, but a relationship matrix in which an expert with knowledge defines the weight may be derived.

[0074] Furthermore, in the above embodiment, the relationship matrix 48 may be made modifiable. In this case, in response to instructions from the input device 15, the display control unit 25 displays a modification screen for the relationship matrix 48 on the display 14 and receives instructions from a user, such as a radiologist, to modify the relationship matrix on the modification screen. Then, the relationship derivation unit 24 modifies the relationship matrix in accordance with the modification instructions.

[0075] Figure 11 shows the modification screen for the relationship matrix. As shown in Figure 11, the modification screen 60 displays the relationship matrix 61. Note that the relationship matrix 61 shown in Figure 11 differs from the relationship matrix 48 shown in Figure 7 in that the values ​​of each element are discretized into five levels. Specifically, the relationships are discretized into five levels in descending order of relationship strength: ++, +, + / -, -, and --. Note that ++, +, + / -, -, and -- can be represented as weight values, for example, 0.20, 0.05, 0.0, -0.05, and -0.20, respectively.

[0076] As shown in Figure 11, for example, for the partially solid characteristic item, the weight for the bronchial radiolucency characteristic item is large (++), while the weight for the solid and ground-glass characteristic items is small (-). The user can modify each element of the relationship matrix 61 on the modification screen 60 to achieve the desired relationship. For example, by selecting the desired element in the relationship matrix 61 using the input device 15, a pull-down menu is displayed in which one of the relationships ++, +, + / -, -, and -- can be selected, and the relationship can be modified by selecting the desired relationship from the displayed pull-down menu.

[0077] By making the elements of the relationship matrix 61 modifiable in this way, the characteristics of the structure of interest can be determined with greater accuracy.

[0078] Furthermore, modifications to the relationship matrix 61 are not limited to selecting discretized relationships from a pull-down menu as described above. The relationship matrix 61 may also be configured to allow the user to select desired elements and input desired weights for those elements as numerical values.

[0079] In the above embodiment, the image analysis unit 22 is provided with a separate extraction model 22A for extracting structures of interest from medical images and a discrimination model 22B for determining the properties of the structures of interest, but it is not limited to this. An extraction model that extracts structures of interest from medical images and also determines the properties of the structures of interest may be used.

[0080] Furthermore, in the above embodiment, the hardware structure of the Processing Unit, which executes various processes such as the information acquisition unit 21, image analysis unit 22, learning unit 23, relationship derivation unit 24, and display control unit 25, can be any of the following types of processors. In addition to a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as described above, the above types of processors can also include FPGAs (Field Programmable Gate Arrays) and other processors whose circuit configuration can be changed after manufacturing. Processors such as Programmable Logic Devices (PLDs) and Application Specific Integrated Circuits (ASICs) are used to execute specific processes. This includes a dedicated electrical circuit, etc., which is a processor with a circuit configuration specifically designed for that purpose.

[0081] A single processing unit may be composed of one of these various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, as is typical of computers such as client and server systems, and this processor functions as multiple processing units. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of a System on a Chip (SoC). Thus, various processing units are configured as hardware structures using one or more of the above-mentioned various processors.

[0082] Furthermore, the hardware structure of these various processors can, more specifically, utilize electrical circuits (Circuitry) that combine circuit elements such as semiconductor devices. . [Explanation of symbols]

[0083] 1. Medical Information System 2. Imaging device 3 Image Interpretation Workshop 4. Medical treatment W S 5 Image Server 6 Image Database 7. Report Server 8 Report Database 10 Networks 11 CPU 12. Information Processing Programs 13 Storage 14 displays 15 Input Devices 16 memory 17 Network Interface 18 bus 20 Information Processing Devices 21 Information Acquisition Department 22 Image Analysis Department 22A Extraction Model 22B Discriminant Model 23 Learning Department 24 Relationship Derivation Unit 25 Display Control Unit 31 CNN 32 Fully connected layer 33 Output Layer 34 Input Layers 35 Convolutional Layers 36 Pooling Layer 37 Fully connected layer 40 Training data 41 Medical Images 42. Structure of Interest Images 43 Properties 44,46 Observations 45,47 Structured labels 48 Relationship Matrix 50 Creation screen 51 Image display area 52 Properties display area 53 Text display area 54. Structure of Concerns 55 marks 56 Confirm button 60 Correction screen 61 Relationship Matrix G0 Medical Images

Claims

1. Equipped with at least one processor, The aforementioned processor, Using a trained neural network, a characteristic score is derived for each of several predetermined characteristic items regarding the structure of interest contained in the image. By analyzing the co-occurrence relationships of descriptions of properties contained in multiple texts, and referring to information representing the relationships between the multiple property items, the property score for at least one of the multiple property items is modified. An information processing device that derives discrimination results for the multiple property items with respect to the structure of interest based on the modified property score.

2. The processor further trains the neural network using training data in which structures of interest contained in medical images and the multiple characteristic items of said structures of interest are identified. The information processing apparatus according to claim 1, which updates information representing the relationship based on the results of the learning.

3. The information processing apparatus according to claim 1 or 2, wherein the information representing the relationship is a relationship matrix in which elements are defined with a larger weight the stronger the co-occurrence relationship between the plurality of characteristic items.

4. The information processing apparatus according to claim 3, wherein the weights are scaled to a predetermined range.

5. The processor presents the matrix of relationships, The information processing device according to claim 3, which modifies the relationship matrix by accepting a modification of the weights in the presented relationship matrix.

6. The aforementioned trained neural network is constructed by machine learning a convolutional neural network. The information processing apparatus according to claim 1 or 2, wherein the processor modifies the characteristic score by a single fully connected layer to which the information representing the relationship is applied and to which the output of the convolutional neural network is input.

7. A computer uses a trained neural network to derive a characteristic score for each of a set of predetermined characteristic items relating to a structure of interest contained in an image, By analyzing the co-occurrence relationships of descriptions of properties contained in multiple texts, and referring to information representing the relationships between the multiple property items, the property score for at least one of the multiple property items is modified. An information processing method for deriving discrimination results for the multiple characteristic items with respect to the interest structure based on the modified characteristic score.

8. A procedure for deriving characteristic scores for each of several predetermined characteristic items regarding structures of interest contained in an image, using a trained neural network, A procedure for correcting the characteristic score for at least one of the multiple characteristic items by referring to information representing the relationships between the multiple characteristic items, which is derived by analyzing the co-occurrence relationships of descriptions of characteristics contained in multiple texts, An information processing program that causes a computer to perform a procedure for deriving discrimination results for the multiple property items with respect to the structure of interest, based on the modified property score.

Images