Method for generating language feature extraction model, information processing device, information processing method and program

Abstract

To provide a method for generating a language feature extraction model capable of extracting a feature amount including a feature of information related to a position in an image from a text related to the image to make the feature amount a feature vector, an information processing device, an information processing method and a program.SOLUTION: In a method for generating a language feature extraction model for making a computer execute processing for extracting a feature from a text related to an image, a system including one or more processors performs machine learning using a plurality of training data including a first image, first position information related to an area of interest in the first image, and a first text explaining the area of interest, inputs the first text to a first model being the language feature extraction model to make the first model output a first feature amount, inputs the first image and the first feature amount to a second model to make the second model estimate the area of interest, and trains the first model and the second model such that the estimated area of interest outputted from the second model coincides with an area of interest of a correct answer shown by the first position information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a language feature extraction model generation method, an information processing device, an information processing method, and a program, and in particular to a natural language processing technology and a machine learning technology that handle text related to an image. [Background technology]

[0002] In recent years, research and development of various types of artificial intelligence (AI) that use text as linguistic information as input has been actively conducted, and commercialization is also progressing. Chatbots and automatic text summarization AI are representative examples. In the case of general AI that obtains a desired output from text input, multiple pairs (data sets) of the text used for input and the correct information to be output when that text is input are prepared, and an AI model is trained using a dataset containing these multiple pairs.

[0003] Non-Patent Document 1 discloses a method for extracting feature amounts from both an image and text, and estimating the relationship between the image and the text.

[0004] Patent document 1 also discloses a slide summarization device that extracts images and text data for each page from slide materials, calculates a score value for each page based on image features for each page calculated based on the amount of extracted image data and text features for that page calculated based on the frequency of occurrence of words contained in the extracted text data, and selects pages from the slide materials so that the total score value of the selected pages is maximized.

[0005] Patent Document 2 discloses a similar image retrieval system including an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, an appearance feature extraction unit that extracts appearance features indicating features of the image's appearance using the appearance information in the image and an appearance feature extraction model, a classification information acquisition unit that acquires classification information indicating the classification of the image, a categorized text feature extraction unit that extracts classified text features indicating features of wording indicating the classification of the image using the classification information in the image and the categorized text feature extraction model, and an overall feature extraction unit that extracts overall features that are features of the entire image in the image, using the appearance features, classified text features, and a multimodal model in the image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-049975 A [Patent Document 2] JP 2021-157570 A [Non-patent literature]

[0007] [Non-Patent Document 1] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He, “Stacked Cross Attention for Image-Text Matching” <https: / / openaccess.thecvf.com / content_ECCV_2018 / papers / Kuang-Huei_Lee_Stacked_Cross_Attention_ECCV_2018_paper.pdf> ,<https: / / arxiv.org / pdf / 1803.08024> Summary of the Invention [Problem to be solved by the invention]

[0008] However, the method described in Non-Patent Document 1 requires a large number of pairs of images including the target region and corresponding text to learn the model. In addition, in recent years, apart from the demand for general AI development, there has been an increasing demand to extract features of text data (language information) and convert them into feature vectors. A feature vector of text is a numerical vector that indicates the characteristics of the text. By converting text into a feature vector, it can be used for various purposes, such as creating an AI that identifies an object in an image that is pointed out by text from an image and text related to that image, or searching for text that contains content similar to a certain text.

[0009] For example, in medical image diagnosis, a large number of radiology reports (text data) containing findings written by doctors after interpreting images taken using CT (Computed Tomography) devices, etc., are stored as past data, and many attempts have been made to utilize such data to assist and streamline the diagnostic work of doctors. If the text, such as findings, contained in such radiology reports can be properly converted into feature vectors, it can be used for various purposes, such as searching for similar past reports or grouping similar reports.

[0010] This is a division of roles for AI, so to speak, and is an AI system that achieves a desired task by combining a feature extraction AI that generates feature vectors from language information with an application-specific AI that receives input of the language feature vectors and performs the desired processing, such as discrimination, classification, or estimation (prediction).To realize such an AI system with divided roles, it is desirable to realize a general-purpose feature extraction AI that generates useful feature vectors that can be used for processing a variety of applications.

[0011] However, when considering a configuration that combines a feature extraction AI with a purpose-specific AI that uses the extracted feature vector to perform the desired processing, whether or not the feature extraction AI realized by machine learning can calculate a valid feature vector is a black box for the AI ​​developer, making it difficult to control. The model created by machine learning depends on the data set used for learning (training). Normally, to increase the versatility of a model, it is necessary to comprehensively prepare a large amount of data that could actually be used as input as learning data.

[0012] In other words, to generate a language feature extraction AI that can output valid language feature vectors that can produce accurate results suited to the final target task, a large number of pairs of text and correct answer data (here, correct answer feature vectors) corresponding to that text are generally required. The mechanism by which language feature extraction AI converts text into feature vectors is a so-called "black box," and it is impossible to explain what criteria are used to calculate what feature vectors, so a large amount of training data is required to create a valid AI.

[0013] On the other hand, it is difficult for humans to prepare correct feature vectors that indicate the characteristics of a certain text as correct data.

[0014] The present disclosure has been made in consideration of the above circumstances, and aims to provide a method for generating a language feature extraction model that is capable of extracting features including information relating to the position of an image from text related to an image and converting the features into feature vectors, an information processing device, an information processing method, and a program. [Means for solving the problem]

[0015] A method for generating a language feature extraction model according to a first aspect of the present disclosure is a method for generating a language feature extraction model that causes a computer to execute a process of extracting features from text related to an image, wherein a system including one or more processors performs machine learning using a plurality of training data including a first image, first location information related to a region of interest in the first image, and first text describing the region of interest, inputs the first text into the first model to output a first feature amount representing a feature of the first text from the first model, inputs the first image and the first feature amount into a second model different from the first model to cause the second model to estimate a region of interest in the first image, and trains the first model and the second model so that the estimated region of interest output from the second model matches a correct region of interest indicated by the first location information, thereby generating a first model that is a language feature extraction model.

[0016] According to the first aspect, the first model is trained to output features including information about the location of a region of interest in an image mentioned in an input text from the input text. That is, the language feature extraction model generated by the first aspect can output features embedded with features about the location of a region of interest in an image from the input text. The features generated by the language feature extraction model can be useful data, for example, in a process of identifying text related to a region of interest in an image or extracting similar text.

[0017] According to the first aspect, when training the first model and the second model, it is not necessary to prepare a correct answer feature amount that is correct data for the output of the first model, and it is possible to make the first model learn the relationship between the text and the position of the region of interest in the image mentioned in the text. According to the first aspect, even when the amount of training data is relatively small, it is possible to generate a high-performance language feature extraction model that can output a feature amount including the feature of the position of the region of interest in the image from the input text. Note that the "model" is actually a program. The method for generating a language feature extraction model is understood as a method for producing a language feature extraction model.

[0018] A method for generating a language feature extraction model according to the second aspect may be configured in such a way that, in the method for generating a language feature extraction model according to the first aspect, a system uses a third model that receives an image feature extracted from an image and a language feature extracted from a text and outputs a degree of relevance between the two, and in machine learning, inputs a second feature extracted from a first image and the first feature into the third model to cause the third model to estimate a degree of relevance between the first image and the first text, and trains the first model and the third model so that the estimated degree of relevance output from the third model matches a correct degree of relevance.

[0019] A method for generating a language feature extraction model according to the third aspect may be configured in such a way that, in the method for generating a language feature extraction model according to the second aspect, a system uses a fourth model that extracts a second feature from an input first image, and in machine learning, inputs the first image and location information to the fourth model to output the second feature, and trains the first model, the third model and the fourth model so that the estimated relevance output from the third model matches the correct relevance.

[0020] A method for generating a language feature extraction model according to a fourth aspect may be configured in such a way that, in the method for generating a language feature extraction model according to the first aspect, a system uses a fifth model that receives an input of language features extracted from each of a plurality of texts and outputs a degree of relevance of the plurality of texts, and in machine learning, inputs a second text different from the first text to the first model, thereby inputting a third feature extracted from the second text by the first model and the first feature into the fifth model to have the fifth model estimate a degree of relevance between the first text and the second text, and training the first model and the fifth model so that the estimated degree of relevance output from the fifth model matches a correct degree of relevance.

[0021] A method for generating a language feature extraction model according to a fifth aspect is a method for generating a language feature extraction model according to any one of the first to fourth aspects, in which the text and the first text may be structured text.

[0022] A sixth aspect of the present invention relates to a method for generating a language feature extraction model in the method for generating a language feature extraction model according to the fourth aspect, wherein the second text is a structured text.

[0023] A method for generating a language feature extraction model relating to the seventh aspect may be configured in such a way that, in the method for generating a language feature extraction model relating to any one of the first to sixth aspects, the system performs a process for displaying a region of interest estimated by the second model.

[0024] A method for generating a language feature extraction model relating to the eighth aspect may be configured such that, in the method for generating a language feature extraction model relating to any one of the first to seventh aspects, the position information includes coordinate information that identifies the position of the region of interest in the first image.

[0025] A method for generating a language feature extraction model according to a 9th aspect is a method for generating a language feature extraction model according to any one of the 1st to 8th aspects, in which the first image may be a cropped image including position information.

[0026] An information processing device according to a tenth aspect includes one or more storage devices in which a program including a language feature extraction model generated by a language feature extraction model generation method according to any one of the first to ninth aspects is stored, and one or more processors that execute the program.

[0027] An information processing device according to an eleventh aspect includes one or more processors and one or more storage devices in which instructions executed by the one or more processors are stored. The one or more processors execute a process of acquiring text describing a region of interest in an image, inputting the text into a first model, and causing the first model to output language features representing characteristics of the text. The first model is a model obtained by machine learning using a plurality of training data including a first image for training, first position information regarding the region of interest in the first image, and the first text describing the region of interest, inputting the first text into the first model and causing the first model to output first features representing characteristics of the first text, inputting the first image and the first features into a second model different from the first model, and causing the second model to estimate the region of interest in the first image, and training the first model and the second model so that the estimated region of interest output from the second model matches the correct region of interest indicated by the first position information.

[0028] An information processing device according to a 12th aspect may be configured in the information processing device described in the 10th or 11th aspect, such that the one or more processors input image features extracted from the second image and language features extracted from the text to a third model, and output a degree of relevance between the second image and the text from the third model.

[0029] An information processing device according to a 13th aspect may be configured in the information processing device according to the 12th aspect, wherein one or more processors acquire a second image and second position information relating to a region of interest in the second image, and input the second image and the second position information to a fourth model, thereby outputting image features from the fourth model.

[0030] An information processing device according to a 14th aspect may be configured in the information processing device according to the 10th or 11th aspect, wherein the one or more processors input linguistic features extracted from each of the multiple texts by the first model to a fifth model, and output the relevance of the multiple texts from the fifth model.

[0031] An information processing device according to a fifteenth aspect is the information processing device according to any one of the tenth to fourteenth aspects, in which the text and the first text may be structured text.

[0032] An information processing method according to a sixteenth aspect includes one or more processors performing a process of acquiring text describing a region of interest in an image, inputting the text into a first model, and causing the first model to output language features representing characteristics of the text, wherein the first model is obtained by machine learning using training data including a first image for training, a first text describing the region of interest in the first image, and first location information related to the region of interest in the first image, inputting the first text into the first model and causing the first model to output first features representing characteristics of the first text, inputting the first image and the first features into a second model different from the first model and causing the second model to estimate the region of interest in the first image, and training the first model and the second model so that the region of interest estimated by the second model matches the region of interest indicated by the first location information.

[0033] The information processing method according to the sixteenth aspect may have a configuration including the same specific aspect as the information processing device according to any one of the second to fifteenth aspects.

[0034] A program according to a seventeenth aspect is a program for causing a computer to realize a function of extracting features from text related to an image, and causes the computer to realize a function of acquiring text describing a region of interest in an image, and a function of inputting the text into a first model and outputting language features representing features of the text from the first model. The first model is a model obtained by machine learning using training data including a first image for training, first position information regarding the region of interest in the first image, and first text describing the region of interest in the first image, inputting the first text into the first model and outputting first features representing features of the first text from the first model, inputting the first image and the first features into a second model different from the first model and causing the second model to estimate the region of interest in the first image, and training the first model and the second model so that the estimated region of interest output from the second model matches the region of interest indicated by the first position information.

[0035] The program according to the seventeenth aspect may have a configuration including the same specific aspect as the information processing device according to any one of the second to fifteenth aspects. Effect of the Invention

[0036] According to the present disclosure, it is possible to generate a language feature extraction model capable of extracting features including features related to the position of a region of interest in an image from text related to the image. The method of generating a language feature extraction model according to the present disclosure does not require providing features as correct answer data in machine learning, and is capable of learning the relationship between text and the position of a region of interest in an image even with a relatively small amount of training data, and is capable of generating a language feature extraction model capable of extracting useful features from input text.

[0037] By using the language feature extraction model generated by the method of the present disclosure, it is possible to provide features that take into account position information in an image. The features generated by the language feature extraction model of the present disclosure can be used for various purposes, such as estimating the correspondence between images and text, and determining the relevance between texts. [Brief description of the drawings]

[0038] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of learning (training) data used in a method for generating a language feature extraction model according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating a schematic functional configuration of the machine learning device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of a hardware configuration of the machine learning device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of a machine learning method executed by the machine learning device according to the first embodiment. [Diagram 5] FIG. 5 is a block diagram showing a schematic functional configuration of a machine learning device using a trained language feature extraction model. [Figure 6] FIG. 6 is a flowchart showing an example of a machine learning method executed by the machine learning device according to the second embodiment. [Figure 7] FIG. 7 is a block diagram illustrating a schematic functional configuration of a machine learning device according to the third embodiment. [Figure 8] FIG. 8 is a block diagram illustrating an example of a hardware configuration of the machine learning device according to the third embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of a machine learning method executed by the machine learning device according to the third embodiment. [Figure 10] FIG. 10 is a block diagram showing a part of the functional configuration of the machine learning device according to the fourth embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of a machine learning method executed by the machine learning device according to the fourth embodiment. [Figure 12] FIG. 12 is a block diagram illustrating a schematic functional configuration of an information processing device according to the fifth embodiment. [Figure 13] FIG. 13 is a block diagram illustrating an example of a hardware configuration of an information processing device according to the fifth embodiment. [Figure 14] FIG. 14 is a block diagram illustrating a schematic functional configuration of an information processing device according to the sixth embodiment. [Figure 15] FIG. 15 is a block diagram illustrating a schematic functional configuration of a machine learning device according to the seventh embodiment. [Figure 16] FIG. 16 is a block diagram illustrating an example of a hardware configuration of the machine learning device according to the seventh embodiment. [Figure 17] FIG. 17 is a flowchart of a machine learning method executed by the machine learning device according to the seventh embodiment. [Figure 18] FIG. 18 is a block diagram illustrating a schematic functional configuration of an information processing device according to the eighth embodiment. [Figure 19] FIG. 19 is a block diagram showing an example of a hardware configuration of an information processing device according to the eighth embodiment. [Figure 20] FIG. 20 is a block diagram illustrating a schematic functional configuration of an information processing device according to the ninth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0040] <<Examples of data used for machine learning>> FIG. 1 is an explanatory diagram showing an example of learning (training) data used in a method for generating a language feature extraction model according to an embodiment of the present disclosure. Here, an example of training data TDj including an image IMj used in medical image diagnosis, position information TPj related to a region of interest ROIj in the image IMj, and a finding sentence TXj described in the region of interest ROIj will be described. Note that "training data" is synonymous with "learning data." The image IMj, the position information TPj related to the region of interest ROIj, and the finding sentence TXj are associated (linked) with each other. The subscript j represents an index number as an identification code of the associated data set. The region of interest ROIj in medical image diagnosis is mainly a lesion area.

[0041] The image IMj may be, for example, a CT image captured using a CT device. FIG. 1 illustrates a CT image obtained by capturing an image of a chest region including the lungs of a subject, but the region to be captured is not limited to the lungs, and may be a region including other organs such as the heart, liver, kidneys, and brain. The imaging device that captures an image of a subject and generates a medical image is not limited to a CT device, and may be another type of modality such as an MRI device, a PET device, or an endoscope device. The image IMj may be a three-dimensional image composed of three-dimensional data obtained by continuously capturing two-dimensional slice tomographic images, or may be a two-dimensional image. The term "image" includes the meaning of image data.

[0042] The position information TPj regarding the region of interest ROIj is information that can specify the position of ROIj in the image IMj. The position information TPj may be coordinate information indicating coordinates in the image IMj, information indicating an area or range in the image IMj, or a combination of these. The position information TPj may be information added as annotation information for the image IMj, or meta information attached to the image IMj, such as a DICOM (Digital Imaging and Communications in Medicine) tag.

[0043] For example, the position information TPj may be coordinate information of the four corners of a rectangle surrounding the range of ROIj, coordinate information of the center of gravity of ROIj, or a segmentation mask image that specifies the area of ​​ROIj in pixel units, etc. Alternatively, when the image IMj itself is a cropped image cut out from the region of interest ROIj, if the image area cut out as the cropped image can be specified, the cropped image itself contains the position information TPj and is understood to be an image IMj equipped with the position information TPj.

[0044] The image IMj is an example of a "first image" in this disclosure, and the position information TPj is an example of a "first position information" in this disclosure.

[0045] The finding sentence TXj may be, for example, a sentence written in a radiology report. The finding sentence TXj is an example of a "first text" in the present disclosure. Here, as the finding sentence TXj, a text that is unstructured data in a free description type sentence format before structuring is exemplified, but structured data that is structured by a structural analysis of the sentence can also be used.

[0046] Such training data TDj can be generated by sampling appropriate data from a database in which data on medical images and interpretation reports relating to past examination cases in a medical institution such as a hospital are associated with each other and stored.

[0047] First embodiment: Example 1 of a method for generating a language feature extraction model [Example of machine learning device configuration] FIG. 2 is a block diagram showing a schematic functional configuration of a machine learning device 10 according to the first embodiment. The machine learning device 10 includes a language feature extraction model 12 which is a first learning model, a domain estimation model 14 which is a second learning model, a loss calculation unit 16, and a parameter update unit 18. The functions of each unit of the machine learning device 10 may be realized by a combination of computer hardware and software. The machine learning device 10 may be configured by a computer system including one or more computers. The machine learning device 10 is an example of a "system" in this disclosure.

[0048] For example, a natural language processing model called BERT (Bidirectional Encoder Representations from Transformers) is applied to the language feature extraction model 12. The language feature extraction model 12 receives an input of a finding sentence TXj, which is a text, extracts a feature amount corresponding to the input finding sentence TXj, and outputs a finding feature LFVj, which is a language feature vector (finding feature vector). The language feature extraction model 12 is an example of a "first model" in the present disclosure. The finding feature LFVj is an example of a "first feature amount" in the present disclosure.

[0049] For example, a convolutional neural network (CNN) is applied to the region estimation model 14. The region estimation model 14 receives an input of an image IMj and a language feature vector LFVj, estimates a lesion region in the image IMj mentioned in the input finding sentence TXj, and outputs estimated region information PAj indicating the position of the estimated lesion region. The estimated region information PAj may be, for example, coordinate information specifying the position of a rectangle (bounding box) surrounding the range of the estimated lesion region, or may be a segmentation mask image specifying the estimated lesion region in pixel units. The region estimation model 14 is an example of a "second model" in the present disclosure. The lesion region indicated by the estimated region information PAj output from the region estimation model 14 is an example of an "estimated region of interest" in the present disclosure.

[0050] The loss calculation unit 16 calculates a loss indicating the error between the estimated lesion area indicated in the estimated area information PAj output from the area estimation model 14 and the correct region of interest ROIj indicated by the correct position information TPj linked to the image IMj.

[0051] Based on the loss calculated by the loss calculation unit 16, the parameter update unit 18 calculates the amount of update for the parameters of each model of the domain estimation model 14 and the language feature extraction model 12 so as to reduce the loss, and updates the parameters of each model according to the calculated amount of update. The parameters of each model include the filter coefficients (weights of connections between nodes) of filters used in processing each layer of the neural network, and biases of the nodes. The parameter update unit 18 optimizes the parameters of each model by a method such as Stochastic Gradient Descent (SGD).

[0052] 3 is a block diagram showing an example of a hardware configuration of the machine learning device 10. The machine learning device 10 includes a processor 102, a computer-readable medium 104 which is a non-transient tangible entity, a communication interface 106, an input / output interface 108, and a bus 110. The processor 102 is connected to the computer-readable medium 104, the communication interface 106, and the input / output interface 108 via the bus 110.

[0053] The form of the machine learning device 10 is not particularly limited, and may be a server, a workstation, a personal computer, or the like.

[0054] The processor 102 includes a central processing unit (CPU). The processor 102 may include a graphics processing unit (GPU). The computer-readable medium 104 includes a memory 112 which is a main storage device and a storage 114 which is an auxiliary storage device. The computer-readable medium 104 may be, for example, a semiconductor memory, a hard disk drive (HDD) device, a solid state drive (SSD) device, or a combination of a plurality of these. The computer-readable medium 104 is an example of a "storage device" in this disclosure.

[0055] The machine learning device 10 may further include an input device 152 and a display device 154. The input device 152 is configured, for example, by a keyboard, a mouse, a multi-touch panel, or other pointing device, or a voice input device, or an appropriate combination of these. The display device 154 is configured, for example, by a liquid crystal display, an organic electro-luminescence (OEL) display, or a projector, or an appropriate combination of these. The input device 152 and the display device 154 are connected to the processor 102 via the input / output interface 108.

[0056] The machine learning device 10 can be connected to an electric communication line (not shown) via the communication interface 106. The electric communication line may be a wide area communication line, a private network communication line, or a combination of these.

[0057] The machine learning device 10 is communicatively connected to an external device such as a training data storage unit 600 via the communication interface 106. The training data storage unit 600 includes a storage in which a training data set including a plurality of training data TDj is stored. Note that the training data storage unit 600 may be built in the storage 114 in the machine learning device 10.

[0058] The computer-readable medium 104 stores a plurality of programs, data, and the like, including a learning processing program 130 and a display control program 140. The term "program" includes the concept of a program module. The processor 102 executes the instructions of the programs stored in the computer-readable medium 104 to function as various processing units.

[0059] The learning process program 130 includes instructions for acquiring training data TDj and executing learning processes for the language feature extraction model 12 and the domain estimation model 14. That is, the learning process program 130 includes a data acquisition program 132, the language feature extraction model 12, the domain estimation model 14, a loss calculation program 136, and an optimizer 138. The data acquisition program 132 includes instructions for executing a process for acquiring training data TDj from the training data storage unit 600.

[0060] The loss calculation program 136 includes an instruction to execute a process of calculating a loss indicating an error between estimated area information indicated by the information indicating the position of the lesion area output from the area estimation model 14 and correct position information TPj corresponding to the finding sentence TXj input to the language feature extraction model 12. The optimizer 138 includes an instruction to execute a process of calculating an update amount of the parameters of each model of the area estimation model 14 and the language feature extraction model 12 from the calculated loss and updating the parameters of each model.

[0061] The display control program 140 generates a display signal required for display output to the display device 154 and includes instructions for executing display control of the display device 154 .

[0062] [Overview of machine learning methods] FIG. 4 is a flowchart showing an example of a machine learning method executed by the machine learning device 10 according to the first embodiment. Before executing the flowchart of FIG. 4, a training dataset is prepared by preparing multiple sets of training data TDj, which is a set of data linked to a training image IMj, a finding statement TXj, which is text explaining a region of interest ROIj in the image IMj, and position information TPj related to the region of interest ROIj.

[0063] In step S100, the processor 102 acquires a data set including an image IMj, position information TPj on a region of interest ROIj in the image IMj, and a comment statement TXj describing the region of interest ROIj from a training data set.

[0064] In step S110, the processor 102 inputs the finding sentence TXj to the language feature extraction model 12, causes the language feature extraction model 12 to extract finding features LFVj indicating the feature amount of the finding sentence TXj, and obtains an output of the finding features LFVj from the language feature extraction model 12. The finding features LFVj are expressed by a language feature vector obtained by making the finding sentence TXj into a feature vector.

[0065] In step S120, the processor 102 inputs the finding feature LFVj output by the language feature extraction model 12 and the image IMj linked to the finding sentence TXj to the region estimation model 14, and causes the region estimation model 14 to estimate a region of interest (lesion region) in the image IMj mentioned in the finding sentence TXj. The region estimation model 14 outputs estimated region information PAj estimated from the input finding feature LFVj and image IMj.

[0066] In step S130, processor 102 calculates a loss indicating an error between estimated region information PAj of the lesion region estimated by region estimation model 14 and position information TPj of the correct region of interest ROIj.

[0067] In step S140, the processor 102 calculates parameter update amounts for each of the language feature extraction model 12 and the domain estimation model 14 so as to minimize the loss.

[0068] Then, in step S150, processor 102 updates the parameters of each model of language feature extraction model 12 and region estimation model 14 according to the calculated parameter update amount. Note that training each model to minimize loss means training each model so that the estimated lesion region estimated by region estimation model 14 matches the correct region of interest ROIj (so that the error between them becomes small). The operations of steps S100 to S150 described above may be performed in mini-batch units.

[0069] After step S150, in step S160, the processor 102 determines whether or not to end the learning. The learning end condition may be determined based on the loss value or the number of parameter updates. As a method based on the loss value, for example, the learning end condition may be that the loss has converged within a specified range. As a method based on the number of updates, for example, the learning end condition may be that the number of updates has reached a specified number. Alternatively, a data set for evaluating the performance of the model may be prepared separately from the training data, and whether or not to end the learning may be determined based on an evaluation value using the evaluation data.

[0070] If the determination result in step S160 is a No determination, the processor 102 returns to step S100 and continues the learning process. On the other hand, if the determination result in step S160 is a Yes determination, the processor 102 ends the flowchart of FIG.

[0071] The thus generated learned (trained) language feature extraction model 12 is a model that can receive an input of a finding sentence and output a finding feature (feature vector) in which information on the position of a lesion area (area of ​​interest) in an image referred to in the finding sentence is embedded. In other words, information necessary for identifying the position of a lesion area in an image is embedded in the finding feature output by the language feature extraction model 12. The machine learning method executed by the machine learning device 10 can be understood as a method for generating a language feature extraction model 12 that outputs a language feature vector including information for identifying the position of a lesion area in an image described in a finding sentence, and is an example of a "method for generating a language feature extraction model" in the present disclosure.

[0072] Second embodiment: Example 1 of use of language feature extraction model Fig. 5 is a block diagram showing a schematic functional configuration of a machine learning device 20 using a trained language feature extraction model 12E. The machine learning device 20 shown in Fig. 5 executes a learning process for generating a cross-modal feature integration model 24 that determines the correspondence between an image having position information related to a region of interest in the image and a finding sentence that describes the region of interest.

[0073] The machine learning device 20 includes a language feature extraction model 12E, an image feature extraction model 22, a cross-modal feature integration model 24, a loss calculation unit 26, and a parameter update unit .

[0074] The training dataset may be the same as the dataset used in the first embodiment. For example, CNN is applied to the image feature extraction model 22. The image feature extraction model 22 receives input of an image IMj and position information TPj related to a region of interest ROIj in the image, and outputs image features IFVj indicating the feature amount of the image IMj. The image features IFVj may be expressed by an image feature vector obtained by converting the image IMj into a feature vector. The image features IFVj may be a feature map of multiple channels.

[0075] The language feature extraction model 12E is a learned model trained to receive an input of a finding sentence TXi and output a corresponding finding feature LFVi. The finding sentence TXi input to the language feature extraction model 12E is not limited to a finding sentence TXj (i=j) linked to an image IMj, and may be a finding sentence (i≠j) not linked to an image IMj.

[0076] The cross-modal feature integration model 24 receives the image feature IFVj and the finding feature LFVj as input, and outputs a relevance score indicating the relevance between the two. The relevance score may be a numerical value indicating the degree of relevance, and may indicate the degree of certainty of the relevance by a numerical value ranging from 0 to 1, with "0" indicating no relevance and "1" indicating relevance.

[0077] The loss calculation unit 26 calculates a loss indicating an error between the relevance score output from the cross-modal feature integration model 24 and the correct relevance score. When a combination of an image IMj and a finding sentence TXi (i=j) linked thereto is input to the image feature extraction model 22 and the language feature extraction model 12E, the correct relevance score may be determined as "1." On the other hand, when a combination of an image IMj and an unrelated finding sentence TXi (i≠j) not linked thereto is input to the image feature extraction model 22 and the language feature extraction model 12E, the correct relevance score may be determined as "0."

[0078] The parameter update unit 28 calculates the update amount of the parameters of each model, the cross-modal feature integration model 24 and the image feature extraction model 22, so as to minimize the loss calculated by the loss calculation unit 26, and updates the parameters of each model according to the calculated update amount.

[0079] The hardware configuration of the machine learning device 20 may be similar to the example shown in FIG. 3, but includes a cross-modal feature integration model 24 instead of the region estimation model 14 in FIG. 3, and the loss function of the loss calculated by the loss calculation program 136 and the model whose parameters are updated by the optimizer 138 are different from those in the example in FIG. 3.

[0080] [Overview of machine learning methods] 6 is a flowchart showing an example of a machine learning method executed by the machine learning device 20 according to the second embodiment. In step S101, the processor 102 acquires a data set of an image IMj, position information TPj on a region of interest ROIj in the image IMj, and an observation sentence TXi explaining (describes) the region of interest ROIi from a training data set. If i=j in the acquired data set at this time, the processor 102 acquires "1" as the correct relevance score, and if i≠j, acquires "0" as the correct relevance score.

[0081] In step S111, the processor 102 inputs the finding sentence TXi to the language feature extraction model 12E, and causes the language feature extraction model 12E to extract the finding feature LFVi.

[0082] In step S112, the processor 102 inputs the image IMj and position information TPj relating to the region of interest ROIj in the image IMj to the image feature extraction model 22, and causes the image feature extraction model 22 to extract image features IFVj.

[0083] In step S114, the processor 102 inputs the image features IFVj output from the image feature extraction model 22 and the finding features LFVi output from the language feature extraction model 12E to the cross-modal feature integration model 24, and causes the cross-modal feature integration model 24 to estimate a relevance score. The processor 102 causes the image feature extraction model 22 to extract the image features IFVj.

[0084] Then, in step S128, the processor 102 calculates a loss indicating an error between the relevance score (estimated value) output from the cross-modal feature integration model 24 and the correct relevance score.

[0085] Then, in step S142, the processor 102 calculates parameter update amounts for each of the image feature extraction model 22 and the cross-modal feature integration model 24 so as to minimize the calculated loss.

[0086] In step S152, the processor 102 updates the parameters of the image feature extraction model 22 and the cross-modal feature integration model 24 in accordance with the calculated parameter update amounts.

[0087] The operations of steps S101 to S152 shown in FIG. 6 may be performed in units of mini-batches.

[0088] After step S152, in step S160, the processor 102 determines whether or not to end the learning.

[0089] If the determination result in step S160 is a No determination, the processor 102 returns to step S101 and continues the learning process. On the other hand, if the determination result in step S160 is a Yes determination, the processor 102 ends the flowchart of FIG.

[0090] By training each model in this way, it is possible to construct an association determination AI that can accurately determine whether an input image and a finding correspond (are related or not).

[0091] Third embodiment: Example 2 of a method for generating a language feature extraction model In the above-mentioned second embodiment, the parameters of the trained language feature extraction model 12E are fixed, but the machine learning method described in the first embodiment and the machine learning method described in the second embodiment may be combined to simultaneously train four models, namely, the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, and the cross-modal feature integration model 24. An example is shown in Figures 7 to 9.

[0092] Fig. 7 is a block diagram showing a schematic functional configuration of a machine learning device 30 according to the third embodiment. In the configuration shown in Fig. 7, elements that are the same as or similar to the configurations shown in Fig. 2 and Fig. 5 are given the same reference numerals, and duplicated explanations will be omitted.

[0093] The machine learning device 30 includes a language feature extraction model 12, a region estimation model 14, an image feature extraction model 22, a cross-modal feature integration model 24, loss calculation units 16 and 26, and a parameter update unit 28A. The cross-modal feature integration model 24 is an example of a "third model" in this disclosure, and the image feature extraction model 22 is an example of a "fourth model" in this disclosure. The image features IFVj output by the image feature extraction model 22 are an example of a "second feature amount" in this disclosure.

[0094] The parameter update unit 28A calculates parameter update amounts for each of the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, and the cross-modal feature integration model 24 based on a third loss obtained by integrating the first loss calculated by the loss calculation unit 16 and the second loss calculated by the loss calculation unit 26, and updates the parameters of each model. The method of integrating the first loss and the second loss may be, for example, the sum, average, or weighted average of the first loss and the second loss.

[0095] In other words, all models are trained so that the outputs of the relevance score estimated by the cross-modal feature integration model 24 and the lesion area (area of ​​interest) estimated by the area estimation model 14 are correct (approach the correct answer).

[0096] The relevance score output from the cross-modal feature integration model 24 is an example of an "estimated relevance" in the present disclosure. Note that, although the loss calculation unit 16 and the loss calculation unit 26 are shown separately in Fig. 7, the loss calculation units 16 and 26 may be a common calculation unit, and may have a calculation function of calculating a third loss by integrating a first loss calculated by the loss calculation unit 16 for the output of the region estimation model 14 and a second loss calculated by the loss calculation unit 26 for the output of the cross-modal feature integration model 24.

[0097] By adopting such a machine learning method and training four models simultaneously, the first loss calculated from the output of the region estimation model 14 and the second loss calculated from the output of the cross-modal feature integration model 24 are each fed back to the training of the language feature extraction model 12 and the image feature extraction model 22, thereby improving the performance of each model.

[0098] According to the third embodiment, features related to the position of the region of interest in the image are embedded in the finding features output from the language feature extraction model 12. By training the cross-modal feature integration model 24 using such finding features, it becomes possible to correctly link (associate) the finding statement with the region of interest (lesion region) in the image that the finding statement describes.

[0099] The configuration shown in FIG. 7 can also be applied to fine-tuning the language feature extraction model 12E that has been trained according to the first embodiment.

[0100] FIG. 8 is a block diagram showing an example of a hardware configuration of a machine learning device 30 according to the third embodiment. The configuration shown in FIG. 8 will be described in terms of differences from FIG. 3. The hardware configuration of the machine learning device 30 may be the same as the example shown in FIG. 3, and includes a learning processing program 230 instead of the learning processing program 130 in FIG. 3. The learning processing program 230 includes an instruction to acquire a data set used for training and execute learning processing for all models, the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, and the cross-modal feature integration model 24. The learning processing program 230 includes a data acquisition program 232, the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, the cross-modal feature integration model 24, a loss calculation program 236, and an optimizer 238.

[0101] The data acquisition program 232 includes an instruction to execute a process of acquiring a training data set from the training data storage unit 600. The loss calculation program 236 includes an instruction to execute a process of calculating a first loss indicating an error between the estimated area information output from the area estimation model 14 and the correct position information TPi, a process of calculating a second loss indicating an error between the relevance score output from the cross-modal feature integration model 24 and the correct relevance score, and a process of integrating the first loss and the second loss to calculate a third loss. The optimizer 238 includes an instruction to execute a process of calculating an update amount of the parameters of each model of the area estimation model 14 and the language feature extraction model 12 from the calculated third loss and updating the parameters of each model. Other configurations may be the same as the configuration of the machine learning device 10 shown in FIG. 3.

[0102] [Overview of machine learning methods] Fig. 9 is a flowchart showing an example of a machine learning method executed by the machine learning device 30 according to the third embodiment. In the flowchart shown in Fig. 9, steps common to the flowcharts shown in Fig. 4 and Fig. 6 are given the same step numbers, and duplicated explanations will be omitted.

[0103] The flowchart shown in FIG. 9 includes steps S112 and S114 between steps S110 and S120 of the flowchart shown in FIG.

[0104] 4. Also, step S128 is included between steps S120 and S130 in FIG. 4, and steps S144 and S154 are included instead of steps S140 and S150 in FIG.

[0105] In step S144, the processor 102 calculates parameter update amounts for each of the image feature extraction model 22, the cross-modal feature integration model 24, the language feature extraction model 12, and the region estimation model 14, based on the combined loss of the loss calculated in step S128 and the loss calculated in step S130, so as to reduce the loss.

[0106] In step S154, the processor 102 updates the parameters of each model according to the calculated parameter update amount. The other steps may be the same as those in FIG.

[0107] [Modification of the third embodiment] As a modified example of the third embodiment, for example, a trained model may be applied to the image feature extraction model 22, and the model may be excluded from the learning process, and parameters of the three models, the language feature extraction model 12, the area estimation model 14, and the cross-modal feature integration model 24, may be updated through learning.

[0108] Fourth embodiment: Example of converting structured text into feature vectors In the above-mentioned first to third embodiments, an example has been described in which a sentence-formatted finding text is used as an input to the language feature extraction models 12 and 12E, but the input to the language feature extraction models 12 and 12E is not limited to a sentence-formatted text, and may be a structured text obtained by a structural analysis of a sentence. The structured text may be, for example, structured data in a CSV (Comma Separated Value) format.

[0109] In the training dataset, instead of or in addition to the finding sentences TXj, structured text (structured findings) may be prepared, or the finding sentences may be subjected to structural analysis and converted into structured data as preprocessing of the input to the language feature extraction models 12 and 12E.

[0110] Fig. 10 is a block diagram showing a part of the functional configuration of the machine learning device 32 according to the fourth embodiment. The machine learning device 32 includes a sentence structure analysis unit 40 as a processing unit that performs pre-processing of an input to the language feature extraction model 12. The sentence structure analysis unit 40 receives an input of a finding sentence TXj in a sentence format, performs a structural analysis of the finding sentence TXj, and generates structured data TSj that structures the finding sentence TXj. Although not shown in Fig. 10, other configurations of the machine learning device 32 may be similar to those of the machine learning device 10, the machine learning device 20, or the machine learning device 30. A sentence structure analysis program is stored in a computer-readable medium 104 of the machine learning device 32.

[0111] [Examples of machine learning methods] Figure 11 is a flowchart showing an example of a machine learning method executed by machine learning device 32. Here, an example of a machine learning method by machine learning device 32 in which the configuration of Figure 10 is added to the configuration of machine learning device 30 described in Figures 7 and 8 is described. In the flowchart shown in Figure 11, steps that are common to the flowchart shown in Figure 9 are given the same step numbers, and duplicated explanations will be omitted.

[0112] 11, steps S102 and S111 are included instead of step S110 in FIG.

[0113] After step S100, in step S102, the processor 102 performs a structural analysis on the sentence-formatted finding sentence TXj, and structures the finding sentence TXj.

[0114] Then, in step S111, the processor 102 inputs the structured text (structured findings) to the language feature extraction model 12 to generate finding features LFVj. The subsequent processing may be similar to the flowchart shown in FIG.

[0115] [Modification of the fourth embodiment] In a training dataset, when structured data TSj corresponding to a finding sentence TXj is prepared in advance, the structured finding (structured data TSj) may be acquired instead of acquiring the finding sentence TXj in step S100 of the flowchart shown in FIG. 9.

[0116] Fifth embodiment: Example 2 of utilization of trained language feature extraction model In the fifth embodiment, an example of an information processing device 50 will be described that uses a language feature extraction model 12, an image feature extraction model 22, and a cross-modal feature integration model 24 trained by the method of the third embodiment to which the configuration of the fourth embodiment is applied.

[0117] FIG. 12 is a block diagram showing a schematic functional configuration of an information processing device 50 according to a fifth embodiment. The information processing device 50 includes a data acquisition unit 52, a sentence structure analysis unit 54, a language feature extractor 13, an image feature extractor 23, a cross-modal feature integrator 25, and a determination result output unit 56. The functions of each unit of the information processing device 50 can be realized by a combination of computer hardware and software. The information processing device 50 may be configured by a computer system including one or more computers. The form of the information processing device 50 is not particularly limited, and may be a server, a workstation, a personal computer, or a tablet terminal. The information processing device 50 may be, for example, a viewer terminal used for image interpretation.

[0118] The data acquisition unit 52 acquires an image IMx to be processed, position information TPx relating to a region of interest ROIx in the image IMx, and a finding statement TXy not linked to the image IMx. These data may be acquired from a data server (not shown) or the like. The image IMx is an example of a "second image" in this disclosure, and the position information TPx is an example of a "second position information" in this disclosure. The finding statement TXy is an example of a "text" in this disclosure.

[0119] The image feature extractor 23 is a processing unit to which the trained image feature extraction model 22 is applied. An image IMx and position information TPx relating to a region of interest ROIx in the image IMx are input to the image feature extractor 23. The image feature extractor 23 receives the image IMx and the position information TPx relating to the region of interest ROIx as input, and outputs an image feature IFVx. The image feature IFVx is an example of an "image feature amount" in this disclosure.

[0120] On the other hand, the finding sentence TXy acquired via the data acquisition unit 52 is input to the sentence structure analysis unit 54 and converted into structured data TSy. The sentence structure analysis unit 54 may be a processing unit similar to the sentence structure analysis unit 40 described in Fig. 40. The sentence structure analysis unit 54 performs a structural analysis of the finding sentence TXy, and outputs structured data TSy which is a structured text (structured finding).

[0121] The language feature extractor 13 is a processing unit to which the trained language feature extraction model 12 is applied. Structured data TSy corresponding to an observation sentence TXy is input to the language feature extractor 13. The language feature extractor 13 receives the input of the structured data TSy and outputs an observation feature LFVy. The prediction feature LFVy is an example of a "language feature" in the present disclosure.

[0122] The thus generated finding feature LFVy and image feature IFVx are input to the cross-modal feature integrator 25. The cross-modal feature integrator 25 is a processing unit to which the trained cross-modal feature integration model 24 is applied. The cross-modal feature integrator 25 receives the finding feature LFVy and the image feature IFVx as input, and determines the relevance between the region of interest ROIx in the image IMx and the finding text TXy. The cross-modal feature integrator 25 may determine the presence or absence of relevance and output a determination result of "relevant" or "not relevant", or may output an evaluation value (relevance score) indicating the degree of relevance.

[0123] The determination result output unit 56 performs a process of outputting the determination result by the cross-modal feature integrator 25. The determination result output unit 56 may be configured to perform at least one process of, for example, displaying the determination result, recording the determination result in a database or the like, printing the determination result, and transmitting the determination result to an external device.

[0124] 13 is a block diagram illustrating an example of a hardware configuration of an information processing device 50. The information processing device 50 includes a processor 502, a computer-readable medium 504, a communication interface 506, an input / output interface 508, and a bus 510. The computer-readable medium 504 includes a memory 512 and a storage 514. The information processing device 50 also includes an input device 552 and a display device 554. These elements in the information processing device 50 may have the same configuration as the corresponding elements of the machine learning device 10 described in FIG. 3.

[0125] The computer-readable medium 504 stores various programs, data, etc., including a data acquisition program 532, a sentence structure analysis program 534, a language feature extraction model 12E, an image feature extraction model 22E, a cross-modal feature integration model 24E, a discrimination result presentation program 536, and a display control program 540.

[0126] The data acquisition program 532 includes instructions for executing a process for acquiring data to be processed. The text structure analysis program 534 includes instructions for executing a process for analyzing the structure of an input text and generating structured text data (structured data).

[0127] The language feature extraction model 12E, the image feature extraction model 22E, and the cross-modal feature integration model 24E are trained models obtained by training the language feature extraction model 12, the image feature extraction model 22, and the cross-modal feature integration model 24 by the methods described in the third and fourth embodiments, respectively.

[0128] The discrimination result presentation program 536 includes an instruction for executing an output process for presenting the discrimination result output from the cross-modal feature integration model 24E.

[0129] The computer-readable medium 504 also includes an analysis information storage area 538 that stores analysis information including structured data that is the analysis result of the text structure analysis program 534. The structured text data may be stored in association with a finding sentence in a text format.

[0130] The information processing device 50 can be connected to a medical image storage unit 610 and a report storage unit 612 via the communication interface 506. The medical image storage unit 610 may be, for example, a storage in a medical image management system such as a PACS (Picture Archiving and Communication Systems). The medical image storage unit 610 may be a DICOM server that stores medical images in accordance with the DICOM standard.

[0131] The report storage unit 612 may be a report storage server that stores and manages image interpretation reports including findings prepared by a doctor in medical image diagnosis, or may be a medical data storage server that has both the functions of the medical image storage unit 610 and the report storage unit 612.

[0132] According to the information processing device 50, it is possible to determine the relevance between a finding sentence that is not linked to an image and an image, and to link the image and the finding sentence that are determined to be related. The processing method executed by the information processing device 50 is an example of an "information processing method" in the present disclosure.

[0133] [Modification 1 of the fifth embodiment] 12, an example in which the linguistic feature extractor 13 receives an input of a structured finding has been described, but the present invention is not limited to this, and the linguistic feature extractor 13 may be configured to receive an input of a finding sentence in a sentence format. In this case, the sentence structure analysis unit 54 in FIG. 12 may be deleted.

[0134] [Modification 2 of the fifth embodiment] Although an example has been described in which area estimation model 14 described in Fig. 7 etc. is used as an auxiliary means for training language feature extraction model 12 and area estimation model 14 is separated after training and the trained language feature extraction model 12 is utilized, it is also possible to combine trained area estimation model 14 with trained language feature extraction model 12 and use it as a lesion area estimation AI, just as during training. This lesion area estimation AI can accept inputs of an image and a finding statement related to the image, and output an estimation result of the lesion area in the image mentioned in the finding statement.

[0135] Sixth embodiment: Third example of utilization of trained language feature extraction model 14 is a block diagram showing a schematic functional configuration of an information processing device 60 according to the sixth embodiment. The information processing device 60 is a device that can perform a process of performing a structure analysis and feature vectorization of a finding sentence described in a radiology report when the report is created, and storing the finding sentence in a sentence format, the structured findings, and the feature vectorized finding features in a linked manner.

[0136] The information processing device 60 includes a data acquisition unit 62, a sentence structure analysis unit 54, a language feature extractor 13, a computer aided diagnosis (CAD) unit 64, and a data storage unit 66. The functions of each unit of the information processing device 60 can be realized by a combination of computer hardware and software. The information processing device 60 may be configured by a computer system including one or more computers.

[0137] The data acquisition unit 62 receives input of a medical image and a commentary to be interpreted. The data acquisition unit 62 may automatically acquire the target data from the medical image storage unit 610 or the report storage unit 612, or may receive the target data based on an instruction from an input device.

[0138] The CAD unit 64 performs image processing on the input medical image to generate CAD information that supports image diagnosis. The CAD unit 64 includes, for example, an organ recognition program and / or a disease detection program. The organ recognition program includes, for example, a processing module that performs organ segmentation. The organ recognition program may include a lung segment labeling program, a blood vessel region extraction program, and a bone labeling program.

[0139] The disease detection program includes a detection processing module corresponding to a specific disease, and may include, for example, at least one of a pulmonary nodule detection program, a pulmonary nodule characterization program, a pneumonia CAD program, a breast CAD program, a liver CAD program, a brain CAD program, and a colon CAD program.

[0140] Such a program for CAD may be an AI processing module that includes a trained model that has been trained to apply machine learning such as deep learning to obtain an output for a desired task.

[0141] The CAD information output from the CAD unit 64 may include, for example, information indicating the position of a lesion area or the like in an image, or information indicating a class classification such as a disease name, or a combination of these.

[0142] The sentence structure analysis unit 54 performs a structural analysis of the finding sentence acquired via the data acquisition unit 52, and generates a structured finding.

[0143] The language feature extractor 13 receives an input of a finding sentence acquired via the data acquisition unit 52 or a structured finding structured by the sentence structure analysis unit 54, and generates a finding feature.

[0144] The information processing device 60 performs processing to associate medical images, CAD information, findings, structured findings, and findings features and store them in the data storage unit 66. The information processing device 60 can build a database in which a large number of such data sets are accumulated in the data storage unit 66.

[0145] Seventh embodiment: Example of application to process of searching for similar findings The finding features generated by the language feature extraction model 12E can also be used to compare finding sentences with each other. In the seventh embodiment, an example is shown in which a system is provided that uses finding features extracted from each of a plurality of finding sentences to determine whether the finding sentences state similar contents (highly related contents) or low related contents (unrelated contents) and searches for candidates of similar finding sentences (related finding sentences) from a database.

[0146] Fig. 15 is a block diagram showing a schematic functional configuration of a machine learning device 70 according to the seventh embodiment. In the configuration shown in Fig. 15, elements that are the same as or similar to the configurations shown in Fig. 2 and Fig. 7 are given the same reference numerals, and duplicated explanations will be omitted.

[0147] The machine learning device 70 includes language feature extraction models 12A and 12B, a domain estimation model 14, a correspondence estimation model 124, loss calculation units 16 and 126, and a parameter update unit 128. For convenience of explanation, two language feature extraction models 12A and 12B are shown in Fig. 15, but these are the same (common) language feature extraction model 12.

[0148] The machine learning device 70 receives input of a plurality of finding sentences TXi and TXk, and inputs each of the received finding sentences TXi and TXk into the language feature extraction models 12A and 12B to generate finding features LFVi and LFVk corresponding to each finding sentence TXi and TXk. The finding sentences TXi and TXk are examples of the "first text" and the "second text" in the present disclosure. The finding features LFVi and LFVk are examples of the "first feature amount" and the "third feature amount" in the present disclosure.

[0149] The correspondence estimation model 124 receives an input of a combination of these multiple finding features LFVi and LFVk, estimates a correspondence between the two, and outputs a relevance score indicating the degree of relevance. The relevance score may be defined as a value such as "1" if there is a correspondence (relevance) between the finding sentences and "0" if there is no correspondence (relevance), or may be configured to take a value ranging from 1 to 0 depending on the degree of relevance. The correspondence estimation model 124 is an example of a "fifth model" in the present disclosure.

[0150] The loss calculation unit 126 calculates a loss (fourth loss) indicating an error between the relevance score output by the correspondence estimation model 124 and the correct relevance score. The correct relevance score is assigned as correct data by evaluating the relevance in advance for a combination of a plurality of finding sentences TXi and TXk used for input. In the case of the two finding sentences TXi and TXk illustrated in FIG. 15, both of them describe contents related to similar lesions and are finding sentences with a high relevance.

[0151] The configurations of the language feature extraction model 12B and the domain estimation model 14, and the configuration of the loss calculation unit 16 and the operations of these units may be similar to the example described in FIG.

[0152] Parameter update unit 128 calculates parameter update amounts for each of correspondence estimation model 124, language feature extraction model 12, and area estimation model 14 based on a fifth loss obtained by integrating the first loss obtained from loss calculation unit 16 and the fourth loss obtained from loss calculation unit 126, and updates the parameters of each model. That is, all models are trained so that the outputs of the relevance score estimated by correspondence estimation model 124 and the lesion area (area of ​​interest) estimated by area estimation model 14 become correct (approach the correct answer).

[0153] In FIG. 15 , the loss calculation unit 16 and the loss calculation unit 126 are shown separately, but the loss calculation units 16 and 126 may be a common calculation unit and may have a calculation function of calculating a fifth loss by integrating a first loss calculated by the loss calculation unit 16 for the output of the area estimation model 14 and a fourth loss calculated by the loss calculation unit 126 for the output of the correspondence estimation model 124.

[0154] Fig. 16 is a block diagram showing an example of a hardware configuration of a machine learning device 70. The hardware configuration of the machine learning device 70 may be the same as that shown in Fig. 8. In the configuration shown in Fig. 16, elements common to the configuration shown in Fig. 8 are given the same reference numerals, and duplicated explanations will be omitted. Regarding the configuration shown in Fig. 16, differences from Fig. 8 will be explained.

[0155] A learning process program 330 is stored in the computer-readable medium 104 of the machine learning device 70, instead of the learning process program 230. The learning process program 330 includes a data acquisition program 332, a language feature extraction model 12, a domain estimation model 14, a correspondence estimation model 124, a loss calculation program 336, and an optimizer 338.

[0156] The data acquisition program 332 includes instructions for executing a process of acquiring a data set including a plurality of finding sentences and corresponding images from the training data storage unit 600. The language feature extraction model 12 includes instructions for executing a process of accepting an input of a combination of the acquired plurality of finding sentences and generating an observation feature for each finding sentence. The loss calculation program 336 includes instructions for executing a process of calculating a fifth loss by integrating a first loss calculated from the output of the region estimation model 14 and a fourth loss calculated from the output of the correspondence estimation model 124.

[0157] The optimizer 338 includes an instruction to calculate, from the calculated fifth loss, update amounts for the parameters of the three models, the language feature extraction model 12, the region estimation model 14, and the correspondence estimation model 124, and to execute a process of updating the parameters of each model. The other configurations may be similar to those in FIG. 8.

[0158] 17 is a flowchart of the machine learning method executed by the machine learning device 70. In step S200, the processor 102 acquires a data set including a plurality of observation sentences TXi, TXk, corresponding images IMi, IMk, and position information TPi, TPk on regions of interest ROIi, ROIk in the images IMi, IMk (i ≠ k).

[0159] In step S210, the processor 102 inputs each of the finding sentences TXi and TXk to the language feature extraction model 12 to generate respective finding features LFVi and LFVk.

[0160] In step S214, the processor 102 inputs each of the finding features LFVi and LFVk to the correspondence estimation model 124, and estimates an association score indicating the association between the two.

[0161] In step S220, the processor 102 inputs the combination of each of the finding features TXi, TXk and the images IMi, IMk to the region estimation model 14 to estimate the lesion region.

[0162] In step S226, the processor 102 calculates a loss indicating an error between the relevance score output from the correspondence estimation model 124 and the relevance score of the outcome.

[0163] In step S230, the processor 102 calculates a loss indicating the error between the position of the lesion area estimated by the area estimation model 14 and the correct position of the region of interest.

[0164] In step S240, the processor 102 calculates parameter update amounts for each of the correspondence estimation model 124, the language feature extraction model 12, and the area estimation model 14 so that the combined loss of the loss calculated in step S226 and the loss calculated in step S230 is reduced.

[0165] In step S254, the processor 102 updates the parameters of each model according to the parameter update amount calculated in step S240. The operations from step S200 to step S254 described above may be performed in mini-batch units.

[0166] After step S254, in step S260, the processor 102 determines whether or not to end the learning. Step S260 may be the same process as step S160 in FIG.

[0167] If the determination result in step S260 is a No determination, the processor 102 returns to step S200. If the determination result in step S260 is a Yes determination, the processor 102 ends the flowchart shown in the figure.

[0168] [Modification of the Seventh Embodiment] 15 and 16 have described an example in which a sentence-formatted finding is input to the language feature extraction model 12. However, as described in the fourth embodiment (FIG. 10), a structured text (structured finding) may be input to the language feature extraction model 12.

[0169] Eighth Embodiment In the eighth embodiment, an example of an information processing device 300 that performs a process of determining a correspondence relationship between finding sentences using a trained language feature extraction model 12E generated by the method of the seventh embodiment will be described.

[0170] 18 is a block diagram showing a schematic functional configuration of an information processing device 300 according to an eighth embodiment. The information processing device 300 includes a data acquisition unit 302, sentence structure analysis units 54A and 54B, language feature extractors 13A and 13B, a correspondence estimator 125, and a determination result output unit 306. The functions of each unit of the information processing device 300 can be realized by a combination of computer hardware and software. The information processing device 300 may be configured by a computer system including one or more computers.

[0171] The data acquisition unit 302 acquires a combination of a plurality of finding sentences TXa and TXb to be compared. The sentence structure analysis unit 54A performs a structural analysis of the finding sentence TXa to generate structured data TSa. Similarly, the sentence structure analysis unit 54B performs a structural analysis of the finding sentence TXb to generate structured data TSb. For convenience of explanation, two sentence structure analysis units 54A and 54B are shown in FIG. 15, but these are the same (common) sentence structure analysis unit 54.

[0172] The language feature extractors 13A and 13B are processing units to which a trained model is applied, in which the language feature extraction model 12 is trained by the machine learning method described in the sixth embodiment. The two language feature extractors 13A and 13B shown in Fig. 15 are the same (common) language feature extractor.

[0173] The language feature extractor 13A receives the structured data TSa and generates corresponding finding features LFVa. Similarly, the language feature extractor 13B receives the structured data TSb and generates corresponding finding features LFVb.

[0174] It is also possible to configure the language feature extractors 13A and 13B to receive the input of the finding sentences TXa and TXb instead of the structured data TSa and TSb and generate the corresponding finding features LFVa and LFVb. In this case, the sentence structure analyzers 54A and 54B may be omitted.

[0175] The correspondence estimator 125 is a processing unit that applies a trained model in which the language feature extractor 13 of the sixth embodiment trains a correspondence estimation model 124 by the machine learning method according to the sixth embodiment. The correspondence estimator 125 receives an input of a combination of finding features LFVa and LFVb, and determines whether or not the two correspond to each other.

[0176] The determination result output unit 306 performs an output process of the determination result of the correspondence output from the correspondence estimator 125. The determination result output unit 306 may output the determination result regarding the presence or absence of a correspondence between two finding sentences, or may generate a list of similar finding sentence candidates using the determination result and output the similar finding sentence candidate list.

[0177] Fig. 19 is a block diagram showing an example of a hardware configuration of the information processing device 300. The hardware configuration of the information processing device 300 may be similar to the example shown in Fig. 13. In the configuration shown in Fig. 19, elements that are the same as or similar to those shown in Fig. 13 are given the same reference numerals, and duplicated explanations will be omitted.

[0178] The computer-readable medium 504 of the information processing device 300 stores a plurality of programs including a data acquisition program 532, a sentence structure analysis program 534, a language feature extraction model 12E, a correspondence estimation model 124E, and a similar finding sentence candidate list generation program 546. The data acquisition program 532 includes an instruction to execute a process of acquiring a finding sentence to be processed. The data acquisition program 532 may acquire data from a database (not shown) in which past reports are stored, or may accept input of data via an input device 552.

[0179] The similar finding sentence candidate list generating program 546 includes an instruction to execute a process of searching for similar finding sentences from a database (not shown) based on the output of the correspondence estimation model 124E and generating a similar finding sentence candidate list including the extracted similar finding sentences.

[0180] Furthermore, the computer-readable medium 504 of the information processing device 300 includes a finding sentence analysis information storage unit 548. The finding sentence analysis information storage unit 548 stores information on the analysis result including the structured data obtained by the sentence structure analysis program 534. Other configurations may be similar to those in FIG.

[0181] Ninth Embodiment In the ninth embodiment, an example of an information processing device 400 that performs a similarity search of finding sentences by utilizing finding features generated using a trained language feature extraction model 12E will be described.

[0182] 20 is a block diagram showing a schematic functional configuration of an information processing device 400 according to the ninth embodiment. The information processing device 400 includes a finding statement receiving unit 402, a language feature extractor 13, a similar search unit 404, and a similar candidate output unit 406. The information processing device 400 may include a database storage unit 650. The database storage unit 650 may be an external device communicably connected to the information processing device 400.

[0183] The functions of each unit of the information processing device 400 can be realized by a combination of computer hardware and software. The information processing device 400 may be configured as a computer system including one or more computers.

[0184] The database storage unit 650 stores a database including a plurality of data sets in which a finding statement FTXj is linked to a finding feature FFVj extracted from the finding statement FTXj.

[0185] In the information processing device 400 of the ninth embodiment, for a large number of finding sentences FTXj included in past reports, a feature vector (finding feature FFVj) is calculated in advance using the language feature extractor 13, and the finding sentences FTXj and the finding feature FFVj are linked and stored in a database.

[0186] Then, the finding sentence receiving unit 402 receives as an input a finding sentence QTx for which similar finding sentences are to be searched, and calculates a finding feature QFv by the language feature extractor 13. The similarity search unit 404 calculates the distance between the vectors of the finding feature QFv and each of the finding features FFVj calculated in advance, and extracts multiple candidates with close distances as similar finding sentence candidates.

[0187] The similar candidate output unit 406 performs an output process of presenting the similar finding sentence candidates extracted by the similarity search unit 404 to the user.

[0188] According to this configuration, candidates of finding sentences similar to the finding sentence QTx received from the finding sentence receiving unit 402 are extracted from the database and presented to the user as a candidate list.

[0189] About the programs that run computers A program that causes a computer to realize some or all of the processing functions of each of the machine learning device 10, machine learning device 20, machine learning device 30, machine learning device 32, machine learning device 70, information processing device 50, information processing device 60, information processing device 300, and information processing device 400 described in each of the above embodiments can be recorded on a computer-readable medium which is a tangible, non-transitory information storage medium such as an optical disk, a magnetic disk, a semiconductor memory, or other such information storage medium, and the program can be provided through this information storage medium.

[0190] In addition, instead of providing the program by storing it on such a tangible, non-transitory computer-readable medium, it is also possible to provide the program signal as a download service using a telecommunications line such as the Internet.

[0191] Furthermore, some or all of the processing functions of each of the above-mentioned devices may be realized by cloud computing, and may also be provided as SaaS (Software as a Service).

[0192] <Hardware configuration of each processing unit> The hardware structure of the processing units that execute various processes, such as the loss calculation unit 16, 26, 126, the parameter update unit 18, 28, 28A, 128, and the sentence structure analysis unit 40 in the machine learning device 10 etc. described in the above-mentioned embodiments, and the data acquisition unit 52, 62, 302, the sentence structure analysis unit 54, the language feature extractor 13, the image feature extractor 23, the cross-modal feature integrator 25, the correspondence estimator 125, the determination result output unit 56, 306, the CAD unit 64, the observation statement receiving unit 402, the similar search unit 404, and the similar candidate output unit 406 in the information processing device 50 etc., is, for example, various processors as shown below.

[0193] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units, GPUs, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration designed specifically to execute specific processes.

[0194] A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, a processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Also, multiple processing units may be composed of one processor. As an example of multiple processing units being composed of one processor, first, as represented by a computer such as a client or a server, there is a form in which one processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system including multiple processing units is used in one IC (Integrated Circuit) chip. In this way, the various processing units are composed of one or more of the above various processors as a hardware structure.

[0195] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0196] Advantages of the embodiments of the present disclosure According to each of the above-described embodiments of the present disclosure, the following effects can be obtained.

[0197] [1] The language feature extraction model 12 is trained to output, from an input finding sentence or structured finding, a finding feature, which is a feature vector including features of the location of the region of interest in the image referred to by the finding sentence or structured finding. The language feature extraction model 12E generated by the method described in the embodiment of the present disclosure can generate a feature vector in which features related to the location of the region of interest in the image are embedded from the input text. The feature vector generated by the language feature extraction model 12E can be used for various purposes, such as a process of determining the relevance between an image and a finding sentence, or a process of searching for similar finding sentences and presenting candidates for similar reports.

[0198] [2] According to the method described in the embodiment of the present disclosure, when training language feature extraction model 12, there is no need to prepare correct features (correct feature vectors) that serve as correct data for the output of language feature extraction model 12. Instead, by using a dataset of image IMj, position information TPj of region of interest ROIj in image IMj, and text of a finding sentence or structured finding describing region of interest ROIj in image IMj, the relationship between the text and the position of the region of interest in the image can be learned.

[0199] [3] According to the method described in the embodiment of the present disclosure, even when the amount of training data is relatively small, a high-performance language feature extraction model 12E can be generated.

[0200] About types of medical images The technology of the present disclosure is not limited to CT images, but can be applied to various medical images taken by various medical devices (modalities), such as MR images taken by an MRI (Magnetic Resonance Imaging) device, ultrasound images projecting human body information, PET images taken by a Positron Emission Tomography (PET) device, and endoscopic images taken by an endoscopic device. Images that are the subject of the technology of the present disclosure are not limited to three-dimensional images, and may be two-dimensional images.

[0201] Other application examples In the above embodiment, images and findings in medical image diagnosis are used as examples, but the scope of application of the present disclosure is not limited to this example, and can be applied to various images and text related to a region of interest in the image, regardless of the purpose. For example, the technology of the present disclosure can be applied to a combination of an image of a structure and text related to a defect in the image.

[0202] "others" The present disclosure is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit and scope of the technical idea of ​​the present disclosure. [Explanation of symbols]

[0203] 10 Machine Learning Device 12, 12A, 12B, 12E Language feature extraction model 13,13A,13B Linguistic feature extractor 14 Area Estimation Model 16 Loss calculation section 18 Parameter Update Section 20 Machine Learning Device 22,22E Image feature extraction model 23 Image Feature Extractor 24,24E Cross-modal feature integration model 25 Cross-modal feature integrator 26 Loss calculation section 28,28A Parameter update section 30,32 Machine learning device 40 Sentence Structure Analysis Department 50 Information processing device 52 Data Acquisition Section 54,54A,54B Sentence structure analysis department 56 Judgment result output unit 60 Information processing equipment 62 Data Acquisition Section 64 CAD Department 66 Data Storage Section 70 Machine Learning Device 102 processors 104 Computer-readable medium 106 Communication Interface 108 Input / Output Interface 110 Bus 112 Memory 114 Storage 124,124E Correspondence estimation model 125 Correspondence Estimator 126 Loss calculation section 128 Parameter Update Section 130 Learning Processing Program 132 Data Acquisition Program 136 Loss Calculation Program 138 Optimizer 140 Display Control Program 152 Input Device 154 Display device 230 Learning Processing Program 232 Data Acquisition Program 236 Loss Calculation Program 238 Optimizer 300 Information processing device 302 Data Acquisition Department 304 Computer-readable medium 306 Judgment result output unit 330 Learning Processing Program 332 Data Acquisition Program 336 Loss Calculation Program 338 Optimizer 400 Information processing device 402 Complaint Reception Department 404 Similarity Search Section 406 Similarity candidate output unit 502 processor 504 Computer-readable medium 506 Communication Interface 508 Input / Output Interface 510 Bus 512 Memory 514 Storage 532 Data Acquisition Program 534 Text Structure Analysis Program 536 Discrimination result display program 538 Analysis information storage area 540 Display Control Program 546 Program for generating a list of similar observation sentence candidates 548 Findings analysis information storage unit 552 Input Device 554 Display device 600 Training Data Storage Unit 610 Medical Image Archives 612 Report Storage Unit 650 Database Storage Unit TDj training data IMi, IMj, IMk, IMx Images ROIi,ROIj,ROIk,ROIx Region of interest TXi,TXj,TXk,TXy,TXa,TXb Observations LFVj, LFVy, LFVa, LFVb Findings and characteristics IFVj IFVx Image Features TPi, TPj, TPk, TPx location information PAj estimated area information TSj, TSy, TSa, TSb structured data FTXj Observations FFVj findings and characteristics QTx findings QFv findings and characteristics S100~S160 Machine learning method steps S200~S260 Machine learning method steps

Claims

1. A method for generating a language feature extraction model that causes a computer to execute a process for extracting features from text related to an image, comprising the steps of: A system including one or more processors, performing machine learning using a plurality of training data including a first image, first position information regarding a region of interest in the first image, and first text describing the region of interest; inputting the first text into a first model and outputting a first feature quantity representing a feature of the first text from the first model; inputting the first image and the first feature amount into a second model different from the first model, and causing the second model to estimate the region of interest in the first image; By training the first model and the second model so that an estimated region of interest output from the second model coincides with the region of interest of a correct answer indicated by the first position information, generating the first model, which is the language feature extraction model; How to generate a language feature extraction model.

2. The system further comprises: a third model that receives an image feature extracted from the image and a language feature extracted from the text and outputs a degree of association therebetween; In the machine learning, a second feature amount extracted from the first image and the first feature amount are input to the third model, and the third model is made to estimate a degree of association between the first image and the first text; training the first model and the third model such that an estimated relevance output from the third model matches a ground truth relevance. The method for generating a language feature extraction model according to claim 1 .

3. The system further comprises: using a fourth model for extracting the second feature amount from the input first image; In the machine learning, The first image and the position information are input to the fourth model, and the fourth model is caused to output the second feature amount; training the first model, the third model, and the fourth model such that the estimated relevance output from the third model matches the correct relevance; The method for generating a language feature extraction model according to claim 2.

4. The system further comprises: a fifth model that receives an input of linguistic features extracted from each of the plurality of texts and outputs a degree of relevance of the plurality of texts; In the machine learning, inputting a second text different from the first text into the first model, thereby extracting a third feature from the second text by the first model, and inputting the first feature into the fifth model, thereby causing the fifth model to estimate a degree of relevance between the first text and the second text; training the first model and the fifth model such that an estimated relevance output from the fifth model matches a correct relevance; The method for generating a language feature extraction model according to claim 1 .

5. the text and the first text are structured texts; A method for generating a language feature extraction model according to any one of claims 1 to 4.

6. the second text is a structured text; The method for generating a language feature extraction model according to claim 4.

7. The system further comprises: performing a process of displaying a region of interest estimated by the second model; The method for generating a language feature extraction model according to claim 1 .

8. the position information includes coordinate information identifying a position of the region of interest in the first image; The method for generating a language feature extraction model according to claim 1 .

9. the first image is a cropped image including the position information; The method for generating a language feature extraction model according to claim 1 .

10. one or more storage devices in which a program including the language feature extraction model generated by the method for generating a language feature extraction model according to claim 1 is stored; one or more processors for executing said programs; An information processing device comprising:

11. one or more processors; one or more memory devices on which instructions are stored for execution by the one or more processors; The one or more processors: Obtaining a text description of a region of interest in the image; A process is executed in which the text is input to a first model and a linguistic feature quantity representing a feature of the text is output from the first model; The first model is By machine learning using a plurality of training data including a first image for training, first position information regarding a region of interest in the first image, and first text describing the region of interest, inputting the first text into the first model and causing the first model to output a first feature amount representing a feature of the first text; inputting the first image and the first feature amount into a second model different from the first model and causing the second model to estimate a region of interest in the first image; a model obtained by training the first model and the second model such that an estimated region of interest output from the second model coincides with a correct region of interest indicated by the first position information; Information processing device.

12. The one or more processors: inputting an image feature extracted from the second image and a linguistic feature extracted from the text into a third model, and outputting a degree of relevance between the second image and the text from the third model; 12. The information processing device according to claim 10 or 11.

13. The one or more processors: obtaining the second image and second location information relating to a region of interest in the second image; inputting the second image and the second position information into a fourth model, thereby outputting the image feature amount from the fourth model; The information processing device according to claim 12.

14. The one or more processors: inputting linguistic features extracted from each of the plurality of texts by the first model into a fifth model, and outputting relevance of the plurality of texts from the fifth model; The information processing device according to claim 10 or 11.

15. the text and the first text are structured texts; The information processing device according to claim 10 or 11.

16. One or more processors Obtaining a text description of a region of interest in the image; A process is executed in which the text is input to a first model and a linguistic feature quantity representing a feature of the text is output from the first model; The first model is by machine learning using training data including a first image for training, a first text describing a region of interest in the first image, and a first position information regarding the region of interest in the first image; inputting the first text into the first model and causing the first model to output a first feature amount representing a feature of the first text; inputting the first image and the first feature amount into a second model different from the first model and causing the second model to estimate a region of interest in the first image; a model obtained by training the first model and the second model such that a region of interest estimated by the second model coincides with a region of interest indicated by the first position information; Information processing methods.

17. A program for causing a computer to realize a function of extracting features from text related to an image, comprising: The computer includes: The ability to obtain text describing areas of interest in an image; a function of inputting the text into a first model and outputting linguistic features representing characteristics of the text from the first model; The first model is By machine learning using training data including a first image for training, first position information regarding a region of interest in the first image, and first text describing the region of interest in the first image, inputting the first text into the first model and causing the first model to output a first feature amount representing a feature of the first text; inputting the first image and the first feature amount into a second model different from the first model and causing the second model to estimate a region of interest in the first image; a model obtained by training the first model and the second model such that an estimated region of interest output from the second model coincides with a region of interest indicated by the first position information; program.

Images