Method for generating a language feature extraction model, information processing device, information processing method, and program

Abstract

To provide a method for generating a language feature extraction model capable of extracting a feature amount including a feature of information related to a position in an image from a text related to the image to make the feature amount a feature vector, an information processing device, an information processing method and a program.SOLUTION: In a method for generating a language feature extraction model for making a computer execute processing for extracting a feature from a text related to an image, a system including one or more processors performs machine learning using a plurality of training data including a first image, first position information related to an area of interest in the first image, and a first text explaining the area of interest, inputs the first text to a first model being the language feature extraction model to make the first model output a first feature amount, inputs the first image and the first feature amount to a second model to make the second model estimate the area of interest, and trains the first model and the second model such that the estimated area of interest outputted from the second model coincides with an area of interest of a correct answer shown by the first position information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a method for generating a language feature extraction model, an information processing apparatus, an information processing method, and a program, and particularly relates to natural language processing technology and machine learning technology for handling text related to images.

Background Art

[0002] In recent years, research and development of various artificial intelligences (AIs) that take text as language information as input have been actively carried out, and productization has also progressed. For example, chatbots or automatic text summarization AIs are typical examples. In the case of a general AI that obtains a desired output for text input, a plurality of pairs (data sets) of the text used for input and the correct answer information that is desired to be output when the text is input are prepared, and the AI model can be learned using a data set including these plurality of pairs.

[0003] Non-Patent Document 1 discloses a method for extracting feature amounts from both an image and text respectively and estimating the relationship between the image and the text.

[0004] Further, Patent Document 1 discloses a slide summarization apparatus that extracts image and text data for each page from slide materials, calculates a score value for each page based on the image feature amount for each page calculated based on the amount of data of the extracted image and the text feature amount of that page calculated based on the appearance frequency of words included in the extracted text data, and selects a page so that the total score value of the pages selected from the slide materials is maximized.

[0005] Patent Document 2 discloses a similar image search system comprising: an appearance information acquisition unit that acquires appearance information indicating the appearance of an image; an appearance feature extraction unit that extracts appearance feature quantities indicating the appearance of an image using the appearance information in the image and an appearance feature extraction model; a classification information acquisition unit that acquires classification information indicating the classification of an image; a classification text feature extraction unit that extracts classification text feature quantities indicating the characteristics of the text indicating the classification of an image using the classification information in the image and a classification text feature extraction model; and a whole feature extraction unit that extracts whole feature quantities which are the characteristics of the entire image in the image using the appearance feature quantities in the image, classification text feature quantities, and a multimodal model. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2017-049975 [Patent Document 2] Japanese Patent Publication No. 2021-157570 [Non-patent literature]

[0007] [Non-Patent Document 1] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He, “Stacked Cross Attention for Image-Text Matching” <https: / / openaccess.thecvf.com / content_ECCV_2018 / papers / Kuang-Huei_Lee_Stacked_Cross_Attention_ECCV_2018_paper.pdf> ,<https: / / arxiv.org / pdf / 1803.08024> [Overview of the project] [Problems that the invention aims to solve]

[0008] However, the method described in Non-Patent Document 1 requires a large number of pairs of images containing the target region and corresponding texts in order to train the model. In recent years, in addition to general AI development requests, there has been an increasing demand to extract features from text data (linguistic information) and convert them into feature vectors. A text feature vector is a numerical vector that represents the features of the text. By converting text into feature vectors, it can be used for various purposes, such as creating an AI that identifies an object in an image that a text refers to from an image and the text about that image, or searching for text that contains similar content to a given text.

[0009] For example, in medical imaging diagnosis, a large amount of historical data has been accumulated from image interpretation reports (text data) containing findings written by physicians after interpreting images taken using CT (Computed Tomography) devices, etc. Many attempts are being made to utilize this data to assist and streamline physicians' diagnostic work. If the text, such as findings, contained in such image interpretation reports can be appropriately converted into feature vectors, it can be used for various purposes such as searching for similar past reports or grouping similar reports.

[0010] This is, so to speak, a division of roles within AI, where a feature extraction AI that generates feature vectors from linguistic information and an application-specific AI that receives linguistic feature vectors as input and performs processing such as discrimination, classification, or estimation (prediction) to achieve the desired task, form an AI system that accomplishes the target task. In order to realize such a division of roles type AI system, it is desirable to realize a general-purpose feature extraction AI that generates useful feature vectors that can be used for processing in various applications.

[0011] However, when considering a configuration that combines a feature extraction AI with application-specific AIs that perform the desired processing using the extracted feature vectors, whether or not the feature extraction AI implemented by machine learning can calculate reasonable feature vectors is a black box for AI developers and difficult to control. The models created by machine learning depend on the dataset used for training. Typically, in order to increase the generality of a model, it is necessary to prepare a large amount of training data that comprehensively covers data that could realistically be used as input.

[0012] In other words, to generate a language feature extraction AI that can output reasonable language feature vectors that produce accurate results aligned with the ultimate task, a large number of pairs of text and corresponding ground truth data (in this case, ground truth feature vectors) are generally required. The mechanism by which language feature extraction AI converts text into feature vectors is a so-called "black box," and it is impossible to explain what criteria are used to calculate what kind of feature vectors are generated. Therefore, a large amount of training data is necessary for the AI ​​to be reasonable.

[0013] On the other hand, it is difficult for humans to prepare ground truth feature vectors that represent the characteristics of a given text as ground truth data.

[0014] This disclosure is made in view of the above circumstances and aims to provide a method for generating a language feature extraction model, an information processing device, an information processing method, and a program that can extract feature quantities containing information about position within an image from text relating to an image and convert them into feature vectors. [Means for solving the problem]

[0015] A method for generating a language feature extraction model according to a first aspect of this disclosure is a method for generating a language feature extraction model that causes a computer to perform a process of extracting features from text associated with an image, wherein a system including one or more processors performs machine learning using a plurality of training data including a first image, first location information relating to a region of interest in the first image, and first text describing the region of interest, inputs the first text into a first model and causes the first model to output a first feature quantity representing the features of the first text, inputs the first image and the first feature quantity into a second model different from the first model and causes the second model to estimate the region of interest in the first image, and trains the first model and the second model so that the estimated region of interest output from the second model matches the correct region of interest indicated by the first location information, thereby generating a first model which is a language feature extraction model.

[0016] According to the first embodiment, the first model is trained to output a feature vector that contains features related to the location of the region of interest in an image mentioned by the input text. That is, the linguistic feature extraction model generated by the first embodiment can output a feature vector embedded with features related to the location of the region of interest in an image from the input text. The feature vector generated by the linguistic feature extraction model can be useful data in processes such as identifying text associated with the region of interest in an image or extracting similar text.

[0017] According to the first embodiment, when training the first and second models, it is not necessary to prepare ground truth features that serve as correct data for the output of the first model, and it is possible to train the first model to learn the relationship between text and the location of the region of interest in the image mentioned in that text. According to the first embodiment, even when the training data is relatively small, it is possible to generate a high-performance language feature extraction model that can output features including the location of the region of interest in the image from the input text. Note that "model" is actually a program. The method for generating a language feature extraction model is understood as a method for producing a language feature extraction model.

[0018] The method for generating a language feature extraction model according to the second embodiment may also include a configuration in which, in the method for generating a language feature extraction model according to the first embodiment, the system uses a third model that receives input of image features extracted from an image and language features extracted from text and outputs the degree of association between the two; and in machine learning, the system inputs a second feature extracted from a first image and a first feature to the third model to estimate the degree of association between the first image and the first text, and trains the first model and the third model so that the estimated degree of association output from the third model matches the correct degree of association.

[0019] The method for generating a language feature extraction model according to the third embodiment may also include a configuration in which, in the method for generating a language feature extraction model according to the second embodiment, the system uses a fourth model that extracts a second feature from an input first image, and in machine learning, the first image and location information are input to the fourth model to cause the fourth model to output the second feature, and the first model, the third model and the fourth model are trained so that the estimated relevance output from the third model matches the correct relevance.

[0020] The method for generating a language feature extraction model according to the fourth embodiment may include a configuration in which, in the method for generating a language feature extraction model according to the first embodiment, the system uses a fifth model that receives input of language features extracted from each of a plurality of texts and outputs the degree of relevance between the plurality of texts, and in machine learning, a second text different from the first text is input to the first model, and the third feature extracted from the second text by the first model and the first feature are input to the fifth model to estimate the degree of relevance between the first text and the second text, and the first model and the fifth model are trained so that the estimated degree of relevance output from the fifth model matches the correct degree of relevance.

[0021] The method for generating a language feature extraction model according to the fifth aspect is the method for generating a language feature extraction model according to any one of the first to fourth aspects, where the text and the first text may be structured text.

[0022] The method for generating a language feature extraction model according to the sixth aspect is the method for generating a language feature extraction model according to the fourth aspect, where the second text may be structured text.

[0023] The method for generating a language feature extraction model according to the seventh aspect is the method for generating a language feature extraction model according to any one of the first to sixth aspects, and the system may be configured to perform a process of displaying a region of interest estimated by the second model.

[0024] The method for generating a language feature extraction model according to the eighth aspect is the method for generating a language feature extraction model according to any one of the first to seventh aspects, and the position information may be configured to include coordinate information for specifying the position of the region of interest in the first image.

[0025] The method for generating a language feature extraction model according to the ninth aspect is the method for generating a language feature extraction model according to any one of the first to eighth aspects, and the first image may be a cropped image including position information.

[0026] The information processing apparatus according to the tenth aspect includes one or more storage devices in which a program including a language feature extraction model generated by the method for generating a language feature extraction model according to any one of the first to ninth aspects is stored, and one or more processors for executing the program.

[0027] The information processing device according to the 11th embodiment comprises one or more processors and one or more storage devices that store instructions executed by the one or more processors, wherein one or more processors acquire text describing a region of interest in an image, input the text to a first model and cause the first model to output linguistic features representing the features of the text, and the first model is obtained by machine learning using a plurality of training data including a first training image, first location information relating to the region of interest in the first image, and a first text describing the region of interest, inputting the first text to the first model and causing the first model to output first features representing the features of the first text, inputting the first image and the first features to a second model different from the first model and causing the second model to estimate the region of interest in the first image, and training the first model and the second model so that the estimated region of interest output from the second model matches the correct region of interest indicated by the first location information.

[0028] The information processing device according to the 12th embodiment may be configured such that, in the information processing device described in the 10th or 11th embodiment, one or more processors input image features extracted from a second image and language features extracted from text to a third model, and the third model outputs the degree of association between the second image and the text.

[0029] The information processing device according to the 13th embodiment may be configured such that, in the information processing device according to the 12th embodiment, one or more processors acquire a second image and second location information relating to a region of interest in the second image, and input the second image and the second location information to a fourth model, thereby causing the fourth model to output image features.

[0030] The information processing device according to the 14th embodiment may be configured such that, in the information processing device according to the 10th or 11th embodiment, one or more processors input language features extracted from each of the multiple texts by the first model to the fifth model, and the fifth model outputs the degree of relevance of the multiple texts.

[0031] The information processing device according to the 15th embodiment is an information processing device according to any one embodiment of the 10th to 14th embodiments, in which the text and the first text may be structured text.

[0032] The information processing method according to the 16th embodiment is a model obtained by having one or more processors acquire text describing a region of interest in an image, input the text into a first model and have the first model output linguistic features representing the features of the text, and by machine learning using training data including a first image for training, a first text describing a region of interest in the first image, and first location information relating to the region of interest in the first image, input the first text into the first model and have the first model output first features representing the features of the first text, input the first image and the first features into a second model different from the first model and have the second model estimate the region of interest in the first image, and training the first model and the second model so that the region of interest estimated by the second model matches the region of interest indicated by the first location information.

[0033] The information processing method relating to the 16th embodiment can be configured to include specific embodiments similar to those of the information processing device relating to any one of the 2nd to 15th embodiments.

[0034] The program according to the 17th embodiment is a program that enables a computer to perform a function of extracting features from text associated with an image, and enables the computer to perform a function of acquiring text that describes a region of interest in an image, and a function of inputting text into a first model and outputting linguistic features representing the features of the text from the first model, wherein the first model is obtained by machine learning using training data including a first image for training, first positional information relating to a region of interest in the first image, and a first text that describes a region of interest in the first image, inputting the first text into the first model and outputting first features representing the features of the first text from the first model, inputting the first image and the first features into a second model different from the first model and having the second model estimate the region of interest in the first image, and training the first model and the second model so that the estimated region of interest output from the second model matches the region of interest indicated by the first positional information.

[0035] The program relating to the 17th aspect may have a configuration that includes specific aspects similar to those of the information processing device relating to any one of the 2nd to 15th aspects. [Effects of the Invention]

[0036] According to this disclosure, it is possible to generate a language feature extraction model that can extract features including features related to the location of a region of interest in an image from text associated with an image. The method for generating a language feature extraction model according to this disclosure does not require providing features as ground truth data in machine learning, and it is possible to learn the relationship between text and the location of a region of interest in an image even with relatively little training data, and it is possible to generate a language feature extraction model that can extract useful features from input text.

[0037] By using the language feature extraction model generated by the method disclosed herein, it becomes possible to provide feature quantities that incorporate positional information within images. The feature quantities generated by the language feature extraction model disclosed herein can be used for various processing applications, such as estimating the correspondence between images and texts, and determining the relationships between texts. [Brief explanation of the drawing]

[0038] [Figure 1] Figure 1 is an explanatory diagram showing an example of training data used in the method for generating a language feature extraction model according to the embodiment of this disclosure. [Figure 2] Figure 2 is a block diagram schematically showing the functional configuration of the machine learning device according to the first embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the hardware configuration of a machine learning device according to the first embodiment. [Figure 4] Figure 4 is a flowchart showing an example of a machine learning method performed by the machine learning device according to the first embodiment. [Figure 5] Figure 5 is a block diagram illustrating the functional configuration of a machine learning system using a pre-trained language feature extraction model. [Figure 6] Figure 6 is a flowchart showing an example of a machine learning method performed by the machine learning device according to the second embodiment. [Figure 7] Figure 7 is a block diagram schematically showing the functional configuration of the machine learning device according to the third embodiment. [Figure 8] Figure 8 is a block diagram showing an example of the hardware configuration of a machine learning device according to the third embodiment. [Figure 9] Figure 9 is a flowchart showing an example of a machine learning method performed by the machine learning device according to the third embodiment. [Figure 10] Figure 10 is a block diagram showing a part of the functional configuration of the machine learning device according to the fourth embodiment. [Figure 11] Figure 11 is a flowchart showing an example of a machine learning method performed by the machine learning device according to the fourth embodiment. [Figure 12] Figure 12 is a block diagram schematically showing the functional configuration of the information processing device according to the fifth embodiment. [Figure 13] Figure 13 is a block diagram schematically showing an example of the hardware configuration of the information processing device according to the fifth embodiment. [Figure 14] Figure 14 is a block diagram schematically showing the functional configuration of the information processing apparatus according to the sixth embodiment. [Figure 15] Figure 15 is a block diagram schematically showing the functional configuration of the machine learning device according to the seventh embodiment. [Figure 16] Figure 16 is a schematic block diagram showing an example of the hardware configuration of a machine learning device according to the seventh embodiment. [Figure 17] Figure 17 is a flowchart of the machine learning method executed by the machine learning device according to the seventh embodiment. [Figure 18] Figure 18 is a block diagram schematically showing the functional configuration of the information processing device according to the eighth embodiment. [Figure 19] Figure 19 is a block diagram showing an example of the hardware configuration of an information processing device according to the eighth embodiment. [Figure 20] Figure 20 is a block diagram schematically showing the functional configuration of the information processing device according to the ninth embodiment. [Modes for carrying out the invention]

[0039] Preferred embodiments of the present invention will be described below with reference to the attached drawings.

[0040] Examples of data used in machine learning Figure 1 is an explanatory diagram showing an example of training data used in the method for generating a language feature extraction model according to the embodiment of this disclosure. Here, an example of training data TDj is described, which includes an image IMj used in medical image diagnosis, location information TPj related to the region of interest ROIj within the image IMj, and a description of the findings TXj. Note that "training data" is synonymous with "learning data". The image IMj, the location information TPj related to the region of interest ROIj, and the findings TXj are related to each other. The subscript j represents the index number as an identification code for the related data set. In medical image diagnosis, the region of interest ROIj is mainly the lesion area.

[0041] Image IMj may be, for example, a CT image taken using a CT scanner. Figure 1 shows an example of a CT image obtained by imaging the chest region including the lungs of a subject, but the area to be imaged is not limited to the lungs, but may include other organs such as the heart, liver, kidneys, and brain. Furthermore, the imaging device that images the subject and generates medical images is not limited to a CT scanner, but may be other types of modalities such as an MRI scanner, PET scanner, or endoscope. Image IMj may be a 3D image composed of 3D data obtained by sequentially acquiring 2D slice tomographic images, or it may be a 2D image. Also, the term "image" includes the meaning of image data.

[0042] Location information TPj for a region of interest (ROIj) is information that can identify the location of the ROIj within the image IMj. Location information TPj may be coordinate information indicating coordinates within the image IMj, information indicating a region or range within the image IMj, or a combination of these. Location information TPj may be information attached as annotation information to the image IMj, or it may be metadata attached to the image IMj, such as a DICOM (Digital Imaging and Communications in Medicine) tag.

[0043] For example, the location information TPj may be the coordinate information of the four corners of a rectangle enclosing the ROIj, the coordinate information of the centroid of the ROIj, or a segmentation mask image that identifies the region of the ROIj on a pixel-by-pixel basis. Alternatively, if the image IMj itself is a cropped image extracted from the region of interest ROIj, then if the image region extracted as a cropped image can be identified, the cropped image itself contains the location information TPj, and is understood to be an image IMj equipped with the location information TPj.

[0044] Image IMj is an example of the “first image” in this disclosure, and location information TPj is an example of the “first location information” in this disclosure.

[0045] The findings document TXj may be, for example, a text written in a radiology report. The findings document TXj is an example of the "first text" in this disclosure. Here, the findings document TXj is exemplified as unstructured data in the form of free-form text before it is structured, but it is also possible to use structured data that has been structured through text structure analysis.

[0046] Such training data TDj can be generated by sampling appropriate data from a database in which medical images and interpretation reports related to past examination cases at medical institutions such as hospitals are linked and stored.

[0047] 《First Embodiment: Example 1 of a Method for Generating a Language Feature Extraction Model》 [Example configuration of a machine learning device] Figure 2 is a block diagram schematically showing the functional configuration of the machine learning apparatus 10 according to the first embodiment. The machine learning apparatus 10 includes a language feature extraction model 12, which is a first learning model, a region estimation model 14, which is a second learning model, a loss calculation unit 16, and a parameter update unit 18. The functions of each part of the machine learning apparatus 10 can be realized by a combination of computer hardware and software. The machine learning apparatus 10 may be configured as a computer system including one or more computers. The machine learning apparatus 10 is an example of a “system” in this disclosure.

[0048] The language feature extraction model 12 applies, for example, a natural language processing model called BERT (Bidirectional Encoder Representations from Transformers). The language feature extraction model 12 accepts an input of observation text TXj, extracts features corresponding to the input observation text TXj, and outputs an observation feature LFVj, which is a language feature vector (observation feature vector). The language feature extraction model 12 is an example of the "first model" in this disclosure. The observation feature LFVj is an example of the "first feature" in this disclosure.

[0049] For example, a Convolutional Neural Network (CNN) is applied to the region estimation model 14. The region estimation model 14 accepts an image IMj and a language feature vector LFVj as inputs, estimates the lesion region in the image IMj mentioned in the input finding text TXj, and outputs estimated region information PAj indicating the location of the estimated lesion region. The estimated region information PAj may be, for example, coordinate information that identifies the location of a bounding box surrounding the estimated lesion region, or it may be a segmentation mask image that identifies the estimated lesion region on a pixel-by-pixel basis. The region estimation model 14 is an example of the "second model" in this disclosure. The lesion region indicated by the estimated region information PAj output from the region estimation model 14 is an example of the "estimated region of interest" in this disclosure.

[0050] The loss calculation unit 16 calculates a loss that represents the error between the estimated lesion region shown in the estimated region information PAj output from the region estimation model 14 and the correct region of interest ROIj shown in the correct location information TPj associated with the image IMj.

[0051] The parameter update unit 18 calculates the amount of parameter updates for each model, the region estimation model 14 and the language feature extraction model 12, based on the loss calculated by the loss calculation unit 16, in order to minimize the loss, and updates the parameters of each model according to the calculated update amount. The parameters of each model include the filter coefficients (weights of connections between nodes) and node biases of the filters used in processing each layer of the neural network. The parameter update unit 18 optimizes the parameters of each model using methods such as stochastic gradient descent (SGD).

[0052] Figure 3 is a block diagram showing an example of the hardware configuration of the machine learning device 10. The machine learning device 10 comprises a processor 102, a computer-readable medium 104 which is a non-temporary tangible object, a communication interface 106, an input / output interface 108, and a bus 110. The processor 102 is connected to the computer-readable medium 104, the communication interface 106, and the input / output interface 108 via the bus 110.

[0053] The form of the machine learning device 10 is not particularly limited and may be a server, a workstation, a personal computer, or the like.

[0054] The processor 102 includes a CPU (Central Processing Unit). The processor 102 may also include a GPU (Graphics Processing Unit). The computer-readable medium 104 includes a memory 112 which is the main memory and a storage 114 which is the auxiliary memory. The computer-readable medium 104 may be, for example, a semiconductor memory, a hard disk drive (HDD), or a solid state drive (SSD), or a combination of these. The computer-readable medium 104 is an example of a “storage device” in this disclosure.

[0055] The machine learning device 10 may further include an input device 152 and a display device 154. The input device 152 is comprised of, for example, a keyboard, mouse, multi-touch panel, or other pointing device, or an audio input device, or an appropriate combination thereof. The display device 154 is comprised of, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, or a projector, or an appropriate combination thereof. The input device 152 and the display device 154 are connected to the processor 102 via an input / output interface 108.

[0056] The machine learning device 10 may be connected to a telecommunications line (not shown) via a communication interface 106. The telecommunications line may be a wide-area communication line, a local area communication line, or a combination of both.

[0057] The machine learning device 10 is connected to an external device, such as a training data storage unit 600, via a communication interface 106. The training data storage unit 600 includes storage where training datasets, including multiple training data TDj, are stored. The training data storage unit 600 may be built on storage 114 within the machine learning device 10.

[0058] The computer-readable medium 104 stores multiple programs and data, including a learning processing program 130 and a display control program 140. The term "program" includes the concept of a program module. The processor 102 functions as various processing units by executing program instructions stored in the computer-readable medium 104.

[0059] The learning processing program 130 includes instructions to acquire training data TDj and to execute the learning process for the language feature extraction model 12 and the region estimation model 14. That is, the learning processing program 130 includes a data acquisition program 132, a language feature extraction model 12, a region estimation model 14, a loss calculation program 136, and an optimizer 138. The data acquisition program 132 includes instructions to execute the process of acquiring training data TDj from the training data storage unit 600.

[0060] The loss calculation program 136 uses the estimated region information indicated by the information showing the location of the lesion region output from the region estimation model 14. PAj The command includes instructions to perform a process that calculates a loss that shows the error between the observation text TXj input to the language feature extraction model 12 and the correct location information TPj. The optimizer 138 includes instructions to calculate the amount of parameter updates for each model, the region estimation model 14 and the language feature extraction model 12, from the calculated loss, and to perform a process that updates the parameters of each model.

[0061] The display control program 140 includes instructions to generate display signals necessary for display output to the display device 154 and to execute display control of the display device 154.

[0062] [Overview of Machine Learning Methods] Figure 4 is a flowchart showing an example of a machine learning method performed by the machine learning device 10 according to the first embodiment. Before executing the flowchart in Figure 4, prepare the training dataset by creating multiple sets of training data TDj, which are sets of data where a training image IMj, an observation text TXj (text describing a region of interest ROIj in the image IMj), and location information TPj related to the region of interest ROIj are linked.

[0063] In step S100, the processor 102 obtains a data set from the training dataset that includes an image IMj, location information TPj relating to the region of interest ROIj in the image IMj, and a description of the ROIj (explanatory text TXj).

[0064] In step S110, the processor 102 inputs the observation sentence TXj into the language feature extraction model 12, causes the language feature extraction model 12 to extract observation features LFVj that represent the features of the observation sentence TXj, and obtains the output of observation features LFVj from the language feature extraction model 12. The observation features LFVj are represented by language feature vectors obtained by feature vectorizing the observation sentence TXj.

[0065] In step S120, the processor 102 inputs the finding feature LFVj output by the language feature extraction model 12 and the image IMj associated with the finding text TXj into the region estimation model 14, causing the region estimation model 14 to estimate the region of interest (lesion region) in the image IMj mentioned in the finding text TXj. The region estimation model 14 outputs estimated region information PAj estimated from the input finding feature LFVj and image IMj.

[0066] In step S130, the processor 102 calculates a loss that represents the error between the estimated region information PAj of the lesion region estimated by the region estimation model 14 and the position information TPj of the ground truth region of interest (ROIj).

[0067] In step S140, the processor 102 calculates the parameter update amounts for each model, the language feature extraction model 12 and the region estimation model 14, in order to minimize the loss.

[0068] Then, in step S150, the processor 102 updates the parameters of the language feature extraction model 12 and the region estimation model 14 according to the calculated parameter update amount. Training each model to minimize loss means training each model so that the estimated lesion region estimated by the region estimation model 14 matches the correct region of interest ROIj (so that the error between the two is small). The operations from steps S100 to S150 described above may be performed in mini-batch units.

[0069] After step S150, in step S160, the processor 102 determines whether or not to terminate the learning process. The learning termination condition may be determined based on the loss value or based on the number of parameter updates. As an example of a method based on the loss value, the learning termination condition may be that the loss has converged to a specified range. As an example of a method based on the number of updates, the learning termination condition may be that the number of updates has reached a specified number. Alternatively, a dataset for evaluating the model's performance may be prepared separately from the training data, and the decision on whether or not to terminate the learning process may be based on evaluation values ​​using the evaluation data.

[0070] If the result of step S160 is No, the processor 102 returns to step S100 and continues the learning process. On the other hand, if the result of step S160 is Yes, the processor 102 terminates the flowchart shown in Figure 4.

[0071] Thus, the generated trained language feature extraction model 12 is a model that, upon receiving a report text as input, can output a report feature (feature vector) in which location information regarding the lesion region (region of interest) in the image mentioned in the report text is embedded. In other words, the report feature output by the language feature extraction model 12 has information embedded in it that is necessary to identify the location of the lesion region in the image. The machine learning method executed by the machine learning device 10 can be understood as a method for generating a language feature extraction model 12 that outputs a language feature vector containing information that identifies the location of the lesion region in the image described in the report text, and is an example of a "method for generating a language feature extraction model" in this disclosure.

[0072] 《Second Embodiment: Example 1 of the Use of a Language Feature Extraction Model》 Figure 5 is a block diagram schematically showing the functional configuration of the machine learning device 20 using the trained language feature extraction model 12E. The machine learning device 20 shown in Figure 5 performs a training process to generate a cross-modal feature integration model 24 that determines the correspondence between an image containing location information about a region of interest in the image and an observational text describing the region of interest.

[0073] The machine learning device 20 includes a language feature extraction model 12E, an image feature extraction model 22, a cross-modal feature integration model 24, a loss calculation unit 26, and a parameter update unit 28.

[0074] The training dataset may be the same as the dataset used in the first embodiment. For example, a CNN may be applied to the image feature extraction model 22. The image feature extraction model 22 accepts an image IMj and location information TPj related to the region of interest (ROIj) within the image as input, and outputs an image feature IFVj that represents the features of the image IMj. The image feature IFVj may be represented by an image feature vector obtained by feature vectorizing the image IMj. The image feature IFVj may also be a multi-channel feature map.

[0075] The linguistic feature extraction model 12E is a pre-trained model that takes an input observation sentence TXi and outputs the corresponding observation feature LFVi. The observation sentence TXi input to the linguistic feature extraction model 12E is not limited to observation sentences TXj (i=j) associated with image IMj, but may also be observation sentences (i≠j) not associated with image IMj.

[0076] The cross-modal feature integration model 24 accepts image features IFVj and findings features LFVj as input and outputs a relevance score indicating the relationship between the two. The relevance score may be a numerical value indicating the degree of relevance; for example, it may be a numerical value in the range of 0 to 1, where "0" indicates no relevance and "1" indicates relevance, indicating the confidence level of the relevance.

[0077] The loss calculation unit 26 calculates a loss that shows the error between the relevance score output from the cross-modal feature integration model 24 and the correct relevance score. When a combination of image IMj and its associated observation text TXi (i=j) is input to the image feature extraction model 22 and the language feature extraction model 12E, the correct relevance score may be set to "1". On the other hand, when a combination of image IMj and an unrelated observation text TXi (i≠j) that is not associated with it is input to the image feature extraction model 22 and the language feature extraction model 12E, the correct relevance score may be set to "0".

[0078] The parameter update unit 28 calculates the amount of parameter updates for each model, the cross-modal feature integration model 24 and the image feature extraction model 22, so as to minimize the loss calculated by the loss calculation unit 26, and updates the parameters of each model according to the calculated update amount.

[0079] The hardware configuration of the machine learning device 20 may be the same as the example shown in Figure 3, but instead of the region estimation model 14 in Figure 3, it includes a cross-modal feature integration model 24, and the loss function of the loss calculated by the loss calculation program 136 and the model whose parameters are updated by the optimizer 138 differ from the example in Figure 3.

[0080] [Overview of Machine Learning Methods] Figure 6 is a flowchart showing an example of a machine learning method executed by the machine learning device 20 according to the second embodiment. In step S101, the processor 102 obtains a data set from the training dataset consisting of an image IMj, location information TPj related to the region of interest ROIj in the image IMj, and an observation statement TXi that describes (is written) the region of interest ROIi. If i=j in the obtained data set, the processor 102 obtains "1" as the correct relevance score, and if i≠j, it obtains "0" as the correct relevance score.

[0081] In step S111, the processor 102 inputs the findings sentence TXi into the language feature extraction model 12E, causing the language feature extraction model 12E to extract the findings features LFVi.

[0082] In step S112, the processor 102 inputs the image IMj and location information TPj related to the region of interest (ROIj) in the image IMj to the image feature extraction model 22, causing the image feature extraction model 22 to extract image features IFVj.

[0083] In step S114, the processor 102 inputs the image feature IFVj output from the image feature extraction model 22 and the finding feature LFVi output from the language feature extraction model 12E into the cross-modal feature integration model 24, causing the cross-modal feature integration model 24 to estimate the relevance score. The image feature extraction model 22 is then used to extract the image feature IFVj.

[0084] Subsequently, in step S128, the processor 102 calculates a loss that represents the error between the relevance score (estimated value) output from the cross-modal feature integration model 24 and the ground truth relevance score.

[0085] Then, in step S142, the processor 102 calculates the parameter update amounts for each model of the image feature extraction model 22 and the cross-modal feature integration model 24 so as to minimize the calculated loss.

[0086] In step S152, the processor 102 updates the parameters of the image feature extraction model 22 and the cross-modal feature integration model 24 according to the calculated parameter update amount.

[0087] The operations in steps S101 to S152 shown in Figure 6 may be performed in units of mini-batches.

[0088] After step S152, in step S160, the processor 102 determines whether or not to terminate the learning process.

[0089] If the result of step S160 is No, the processor 102 returns to step S101 and continues the learning process. On the other hand, if the result of step S160 is Yes, the processor 102 terminates the flowchart shown in Figure 6.

[0090] By training each model in this way, it is possible to construct a relevance determination AI that can accurately determine whether an input image corresponds to (is related to) a given observation text.

[0091] 《Third Embodiment: Example 2 of a Method for Generating a Language Feature Extraction Model》 In the second embodiment described above, the parameters of the trained language feature extraction model 12E were fixed. However, a configuration may be adopted in which the machine learning methods described in the first embodiment and the machine learning methods described in the second embodiment are combined to train four models simultaneously: the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, and the cross-modal feature integration model 24. An example of this is shown in Figures 7 to 9.

[0092] Figure 7 is a block diagram schematically showing the functional configuration of the machine learning device 30 according to the third embodiment. In the configuration shown in Figure 7, elements that are the same as or similar to those shown in Figures 2 and 5 are denoted by the same reference numerals, and redundant explanations are omitted.

[0093] The machine learning device 30 includes a language feature extraction model 12, a region estimation model 14, an image feature extraction model 22, a cross-modal feature integration model 24, loss calculation units 16 and 26, and a parameter update unit 28A. The cross-modal feature integration model 24 is an example of the “third model” in this disclosure, and the image feature extraction model 22 is an example of the “fourth model” in this disclosure. The image feature IFVj output by the image feature extraction model 22 is an example of the “second feature” in this disclosure.

[0094] The parameter update unit 28A calculates the parameter update amount for each model of the language feature extraction model 12, region estimation model 14, image feature extraction model 22, and cross-modal feature integration model 24 based on a third loss obtained by integrating the first loss calculated by the loss calculation unit 16 and the second loss calculated by the loss calculation unit 26, and updates the parameters of each model. The method for integrating the first loss and the second loss may be, for example, the sum, average, or weighted average of the first loss and the second loss.

[0095] In other words, all models are trained so that the relevance score estimated by the cross-modal feature integration model 24 and the output of the lesion region (region of interest) estimated by the region estimation model 14 become correct (approach the correct answer).

[0096] The relevance score output from the cross-modal feature integration model 24 is an example of the "estimated relevance" in this disclosure. Although Figure 7 shows the loss calculation unit 16 and the loss calculation unit 26 separately, the loss calculation units 16 and 26 may be a common calculation unit, and may have a calculation function that calculates a third loss by integrating a first loss calculated by the loss calculation unit 16 for the output of the region estimation model 14 and a second loss calculated by the loss calculation unit 26 for the output of the cross-modal feature integration model 24.

[0097] By employing this machine learning method and simultaneously training four models, the first loss calculated from the output of the region estimation model 14 and the second loss calculated from the output of the cross-modal feature integration model 24 are fed back into the training of the language feature extraction model 12 and the image feature extraction model 22, thereby improving the performance of each model.

[0098] According to the third embodiment, since features relating to the location of the region of interest in the image are embedded in the findings features output from the language feature extraction model 12, by training the cross-modal feature integration model 24 using these findings features, it becomes possible to correctly link (associate) the findings text with the region of interest (pathological region) in the image that the findings text describes.

[0099] Furthermore, the configuration shown in Figure 7 can also be applied when fine-tuning the language feature extraction model 12E that has been trained according to the first embodiment.

[0100] Figure 8 is a block diagram showing an example of the hardware configuration of a machine learning device 30 according to the third embodiment. The differences between the configuration shown in Figure 8 and that shown in Figure 3 will be explained. The hardware configuration of the machine learning device 30 may be the same as the example shown in Figure 3, but instead of the learning processing program 130 in Figure 3, it includes a learning processing program 230. The learning processing program 230 includes instructions to acquire the data set to be used for training and to execute the learning process for all models: the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, and the cross-modal feature integration model 24. The learning processing program 230 includes a data acquisition program 232, the language feature extraction model 12, the region estimation model 14, the image feature extraction model 22, the cross-modal feature integration model 24, the loss calculation program 236, and the optimizer 238.

[0101] The data acquisition program 232 includes an instruction to execute a process to acquire training data sets from the training data storage unit 600. The loss calculation program 236 uses the estimated region information output from the region estimation model 14 and the ground truth position information TP jThe optimizer 238 includes instructions to perform the following: a process to calculate a first loss that shows the error between the relevant score output from the cross-modal feature integration model 24 and the ground truth relevance score; and a process to perform the following: a process to calculate a third loss by integrating the first and second losses. The optimizer 238 includes instructions to perform the process of calculating the amount of parameter updates for each model, the region estimation model 14 and the language feature extraction model 12, from the calculated third loss, and updating the parameters of each model. The other configurations may be the same as those of the machine learning device 10 shown in Figure 3.

[0102] [Overview of Machine Learning Methods] Figure 9 is a flowchart showing an example of a machine learning method executed by the machine learning device 30 according to the third embodiment. In the flowchart shown in Figure 9, steps common to the flowcharts shown in Figures 4 and 6 are given the same step numbers, and redundant explanations are omitted.

[0103] The flowchart shown in Figure 9 includes steps S112 and S114 between steps S110 and S120 of the flowchart shown in Figure 4.

[0104] Furthermore, step S128 is included between steps S120 and S130 in Figure 4, and steps S144 and S154 are included instead of steps S140 and S150 in Figure 4.

[0105] In step S144, the processor 102 calculates the parameter update amounts for each model—the image feature extraction model 22, the cross-modal feature integration model 24, the language feature extraction model 12, and the region estimation model 14—based on the loss obtained by integrating the loss calculated in step S128 and the loss calculated in step S130, in order to minimize the loss.

[0106] In step S154, the processor 102 updates the parameters of each model according to the calculated parameter update amount. The other steps may be the same as in Figure 4.

[0107] [Modified example of the third embodiment] As a variation of the third embodiment, for example, the image feature extraction model 22 can be excluded from training by applying a pre-trained model, while the parameters of the three models—the language feature extraction model 12, the region estimation model 14, and the cross-modal feature integration model 24—can be updated through training.

[0108] 《Fourth Embodiment: An Example of Feature Vectorization of Structured Text》 In the first to third embodiments described above, examples were given in which the text of a written observation is used as input to the language feature extraction models 12 and 12E. However, the input to the language feature extraction models 12 and 12E is not limited to written text, but may also be structured text obtained by structural analysis of the text. The structured text may be, for example, structured data in CSV (Comma Separated Value) format.

[0109] In the training dataset, structured text (structured findings) may be provided instead of, or in addition to, the findings text TXj, or the structure of the findings may be analyzed and converted into structured data as preprocessing for the input to the language feature extraction models 12 and 12E.

[0110] Figure 10 is a block diagram showing a part of the functional configuration of the machine learning device 32 according to the fourth embodiment. The machine learning device 32 includes a document structure analysis unit 40 as a processing unit that performs preprocessing of input to the language feature extraction model 12. The document structure analysis unit 40 receives input of an observation text TXj in sentence form, performs structural analysis of the observation text TXj, and generates structured data TSj which is a structured version of the observation text TXj. Although not shown in Figure 10, other configurations of the machine learning device 32 may be the same as those of the machine learning device 10, machine learning device 20, or machine learning device 30. The computer-readable medium 104 of the machine learning device 32 stores a document structure analysis program.

[0111] [Examples of machine learning methods] Figure 11 is a flowchart illustrating an example of a machine learning method performed by the machine learning device 32. Here, we describe an example of a machine learning method performed by the machine learning device 32, which adds the configuration shown in Figure 10 to the configuration of the machine learning device 30 described in Figures 7 and 8. In the flowchart shown in Figure 11, steps common to the flowchart shown in Figure 9 are given the same step numbers, and redundant explanations are omitted.

[0112] In Figure 11, steps S102 and S111 are included instead of step S110 in Figure 9.

[0113] After step S100, in step S102, the processor 102 performs structural analysis on the documentary-style observation text TXj and structures the observation text TXj.

[0114] Subsequently, in step S111, the processor 102 processes the structured text (structured findings). ) The data is input into the language feature extraction model 12 to generate the observation feature LFVj. Subsequent processing may be the same as the flowchart shown in Figure 9.

[0115] [Modification of the fourth embodiment] If structured data TSj corresponding to the observation text TXj is already provided in the training dataset, then in step S100 of the flowchart shown in Figure 9, instead of obtaining the observation text TXj, you can obtain the structured observation (structured data TSj).

[0116] 《Fifth Embodiment: Example 2 of Utilizing a Trained Language Feature Extraction Model》 In the fifth embodiment, an example of an information processing device 50 is described that uses a language feature extraction model 12, an image feature extraction model 22, and a cross-modal feature integration model 24, which were learned by the method of the third embodiment, applying the configuration of the fourth embodiment.

[0117] Figure 12 is a block diagram schematically showing the functional configuration of the information processing device 50 according to the fifth embodiment. The information processing device 50 includes a data acquisition unit 52, a document structure analysis unit 54, a language feature extractor 13, an image feature extractor 23, a cross-modal feature integrator 25, and a judgment result output unit 56. The functions of each part of the information processing device 50 can be realized by a combination of computer hardware and software. The information processing device 50 may be configured as a computer system including one or more computers. The form of the information processing device 50 is not particularly limited and may be a server, a workstation or personal computer, or a tablet terminal. The information processing device 50 may be, for example, a viewer terminal used for image interpretation.

[0118] The data acquisition unit 52 acquires the image IMx to be processed, location information TPx relating to the region of interest ROIx within the image IMx, and observation text TXy not associated with the image IMx. This data may be acquired from a data server or the like (not shown). Image IMx is an example of the "second image" in this disclosure, location information TPx is an example of the "second location information" in this disclosure, and observation text TXy is an example of the "text" in this disclosure.

[0119] The image feature extractor 23 is a processing unit to which a trained image feature extraction model 22 is applied. The image IMx and location information TPx relating to the region of interest ROIx in the image IMx are input to the image feature extractor 23. The image feature extractor 23 receives the image IMx and location information TPx relating to the region of interest ROIx as inputs and outputs an image feature IFVx. The image feature IFVx is an example of an "image feature" in this disclosure.

[0120] Meanwhile, the observation text TXy acquired via the data acquisition unit 52 is input to the text structure analysis unit 54 and converted into structured data TSy. The text structure analysis unit 54 is shown in Figure 1The processing unit may be similar to the text structure analysis unit 40 described in 0. The text structure analysis unit 54 performs a structural analysis of the observation text TXy and outputs structured data TSy, which is structured text (structured observation).

[0121] The language feature extractor 13 is a processing unit to which a trained language feature extraction model 12 is applied. Structured data TSy corresponding to the observation text TXy is input to the language feature extractor 13. The language feature extractor 13 receives the structured data TSy as input and outputs the observation feature LFVy. place The LFVy feature is an example of a "linguistic feature" in this disclosure.

[0122] The resulting findings feature LFVy and image feature IFVx are input to the cross-modal feature integrator 25. The cross-modal feature integrator 25 is a processing unit to which the trained cross-modal feature integration model 24 is applied. The cross-modal feature integrator 25 receives the findings feature LFVy and image feature IFVx as input and determines the relationship between the region of interest ROIx in the image IMx and the findings text TXy. The cross-modal feature integrator 25 may output a determination result of "relevant" or "not relevant" after determining whether there is a relationship, or it may output an evaluation value (relevance score) indicating the degree of relevance.

[0123] The judgment result output unit 56 performs the process of outputting the judgment result from the cross-modal feature integrator 25. The judgment result output unit 56 may be configured to perform at least one of the following processes: displaying the judgment result, recording the judgment result in a database or the like, printing the judgment result, and transmitting the judgment result to an external device.

[0124] Figure 13 is a block diagram schematically showing an example of the hardware configuration of the information processing device 50. The information processing device 50 comprises a processor 502, a computer-readable medium 504, a communication interface 506, an input / output interface 508, and a bus 510. The computer-readable medium 504 includes a memory 512 and a storage 514. The information processing device 50 also comprises an input device 552 and a display device 554. These elements in the information processing device 50 may have a configuration similar to the corresponding elements of the machine learning device 10 described in Figure 3.

[0125] The computer-readable medium 504 stores various programs and data, including a data acquisition program 532, a text structure analysis program 534, a language feature extraction model 12E, an image feature extraction model 22E, a cross-modal feature integration model 24E, a discrimination result presentation program 536, and a display control program 540.

[0126] The data acquisition program 532 includes instructions to execute a process to acquire the data to be processed. The text structure analysis program 534 includes instructions to execute a process to analyze the structure of the input text and generate structured text data (structured data).

[0127] The language feature extraction model 12E, the image feature extraction model 22E, and the cross-modal feature integration model 24E are each trained models obtained by training the language feature extraction model 12, the image feature extraction model 22, and the cross-modal feature integration model 24 using the methods described in the third and fourth embodiments.

[0128] The discrimination result presentation program 536 includes an instruction to perform an output process that presents the discrimination result output from the cross-modal feature integration model 24E.

[0129] Furthermore, the computer-readable medium 504 includes an analysis information storage area 538 that stores analysis information, including structured data which is the analysis result of the text structure analysis program 534. The structured text data may be stored in association with the observation text in sentence form.

[0130] The information processing device 50 may be connected to the medical image storage unit 610 and the report storage unit 612 via the communication interface 506. The medical image storage unit 610 may be, for example, storage in a medical image management system such as PACS (Picture Archiving and Communication Systems). The medical image storage unit 610 may also be a DICOM server that stores medical images in accordance with the DICOM standard.

[0131] The report storage unit 612 may be a report storage server that stores and manages image interpretation reports, including findings written by physicians in medical image diagnosis. Alternatively, it may be a medical data storage server that combines the functions of both a medical image storage unit 610 and a report storage unit 612.

[0132] According to the information processing device 50, it becomes possible to determine the relationship between observation texts that are not linked to images and images, and to link the images and observation texts that are determined to be related. The processing method performed by the information processing device 50 is an example of the "information processing method" in this disclosure.

[0133] [Modification 1 of the Fifth Embodiment] Figure 12 illustrates an example in which the language feature extractor 13 accepts structured findings as input. However, the language feature extractor 13 is not limited to this configuration and may also accept findings in sentence form as input. In this case, the sentence structure analysis unit 54 in Figure 12 may be omitted.

[0134] [Modification 2 of the Fifth Embodiment] The region estimation model 14, as explained in Figure 7, is used as an auxiliary means for training the language feature extraction model 12. After training, the region estimation model 14 is separated and an example of utilizing the trained language feature extraction model 12 is described. However, similar to the training process, it is also possible to use the trained region estimation model 14 in combination with the trained language feature extraction model 12 as a lesion region estimation AI. This lesion region estimation AI accepts an image and a report text related to the image as input and can output the estimation result of the lesion region in the image mentioned in the report text.

[0135] 《Sixth Embodiment: Example 3 of the Use of a Trained Language Feature Extraction Model》 Figure 14 is a block diagram schematically showing the functional configuration of the information processing device 60 according to the sixth embodiment. The information processing device 60 is a device that, when an image interpretation report is created, performs structural analysis and feature vectorization of the findings text described in the report, and processes the linking and saving of the text-format findings text, the structured findings, and the feature vectorized findings features.

[0136] The information processing device 60 includes a data acquisition unit 62, a document structure analysis unit 54, a language feature extractor 13, a computer-aided diagnosis (CAD) unit 64, and a data storage unit 66. The functions of each part of the information processing device 60 can be realized by a combination of computer hardware and software. The information processing device 60 may be configured as a computer system including one or more computers.

[0137] The data acquisition unit 62 receives input of medical images and findings to be read. The data acquisition unit 62 may automatically acquire the target data from the medical image storage unit 610 or the report storage unit 612, or it may accept the target data based on instructions from the input device.

[0138] The CAD unit 64 performs image processing on the input medical image and generates CAD information to support image diagnosis. The CAD unit 64 is configured to include, for example, an organ recognition program and / or a disease detection program. The organ recognition program includes, for example, a processing module that performs organ segmentation. The organ recognition program may include a lung segment labeling program, a vascular region extraction program and a bone labeling program, etc.

[0139] The disease detection program includes a detection processing module corresponding to a specific disease. For example, the disease detection program may include at least one of the following programs: a pulmonary nodule detection program, a pulmonary nodule characterization program, a pneumonia CAD program, a breast CAD program, a liver CAD program, a brain CAD program, and a colon CAD program.

[0140] Such a CAD program may be an AI processing module that includes a trained model that has been trained to obtain the output for the target task by applying machine learning such as deep learning.

[0141] The CAD information output from the CAD unit 64 may include, for example, information indicating the location of lesion areas within the image, or information indicating a classification such as a disease name, or a combination of these.

[0142] The text structure analysis unit 54 performs structural analysis of the observation text acquired via the data acquisition unit 52 and generates structured observations.

[0143] The language feature extractor 13 receives input from observation texts acquired via the data acquisition unit 52, or structured observations structured by the text structure analysis unit 54, and generates observation features.

[0144] The information processing device 60 performs the process of associating medical images, CAD information, findings text, structured findings, and findings features and storing them in the data storage unit 66. The information processing device 60 can construct a database by accumulating a large number of such data sets in the data storage unit 66.

[0145] 《Seventh Embodiment: Example of Use in Searching for Similar Findings》 The observation features generated by the linguistic feature extraction model 12E can also be used to compare observation sentences with each other. In the seventh embodiment, an example is shown of a system that uses observation features extracted from each of several observation sentences to determine whether the observation sentences describe similar content (highly relevant content) or low-relevance (unrelated content), and searches for similar observation sentence candidates (related observation sentences) from a database.

[0146] Figure 15 is a block diagram schematically showing the functional configuration of the machine learning device 70 according to the seventh embodiment. In the configuration shown in Figure 15, elements that are the same as or similar to those shown in Figures 2 and 7 are denoted by the same reference numerals, and redundant explanations are omitted.

[0147] The machine learning device 70 includes language feature extraction models 12A and 12B, a region estimation model 14, a correspondence estimation model 124, loss calculation units 16 and 126, and a parameter update unit 128. In Figure 15, for the sake of explanation, two language feature extraction models 12A and 12B are shown, but these are the same (common) language feature extraction model 12.

[0148] The machine learning device 70 receives multiple observation sentences TXi and TXk as inputs, and inputs each of the received observation sentences TXi and TXk into language feature extraction models 12A and 12B to generate observation features LFVi and LFVk corresponding to each observation sentence TXi and TXk. The observation sentences TXi and TXk are examples of the "first text" and "second text" in this disclosure. The observation features LFVi and LFVk are examples of the "first feature" and "third feature" in this disclosure.

[0149] The correspondence estimation model 124 accepts input of combinations of these multiple observation features LFVi and LFVk, estimates the correspondence between them, and outputs a relevance score indicating the degree of relevance. The relevance score may be defined as a value such as "1" if there is a correspondence (relevance) between the observation sentences, and "0" if there is no correspondence, and may be configured to take values ​​in the range of 1 to 0 depending on the degree of relevance. The correspondence estimation model 124 is an example of the "fifth model" in this disclosure.

[0150] The loss calculation unit 126 calculates a loss (fourth loss) that shows the error between the relevance score output by the correspondence estimation model 124 and the correct relevance score. The correct relevance score is assigned as correct data by pre-evaluating the relevance of multiple combinations of findings sentences TXi and TXk used as input. In the case of the two findings sentences TXi and TXk exemplified in Figure 15, both describe similar lesions and are findings sentences with a high degree of relevance.

[0151] The configuration of the language feature extraction model 12B and the region estimation model 14, as well as the configuration of the loss calculation unit 16 and the operation of each of these units, may be the same as in the example described in Figure 7.

[0152] The parameter update unit 128 calculates the parameter update amount for each model—the correspondence estimation model 124, the language feature extraction model 12, and the region estimation model 14—based on a fifth loss obtained by integrating the first loss obtained from the loss calculation unit 16 and the fourth loss obtained from the loss calculation unit 126, and updates the parameters of each model. In other words, all models are trained so that the relevance score estimated by the correspondence estimation model 124 and the output of the lesion region (region of interest) estimated by the region estimation model 14 become correct (approach the correct answer).

[0153] Although Figure 15 shows the loss calculation unit 16 and the loss calculation unit 126 separately, the loss calculation units 16 and 126 may be a common calculation unit, and may have a calculation function that integrates the first loss calculated by the loss calculation unit 16 for the output of the domain estimation model 14 and the fourth loss calculated by the loss calculation unit 126 for the output of the correspondence estimation model 124 to calculate a fifth loss.

[0154] Figure 16 is a block diagram schematically showing an example of the hardware configuration of the machine learning device 70. The hardware configuration of the machine learning device 70 may be the same as that of Figure 8. For the configuration shown in Figure 16, elements common to the configuration shown in Figure 8 are denoted by the same reference numerals, and redundant explanations are omitted. The differences between the configuration shown in Figure 16 and Figure 8 will be explained below.

[0155] The computer-readable medium 104 of the machine learning device 70 stores a learning processing program 330 instead of the learning processing program 230. The learning processing program 330 includes a data acquisition program 332, a language feature extraction model 12, a region estimation model 14, a correspondence estimation model 124, a loss calculation program 336, and an optimizer 338.

[0156] The data acquisition program 332 includes an instruction to execute a process to acquire a data set containing multiple observation sentences and corresponding images from the training data storage unit 600. The language feature extraction model 12 includes an instruction to accept the input of the acquired combination of multiple observation sentences and to execute a process to generate observation features for each observation sentence. The loss calculation program 336 includes an instruction to execute a process to calculate a fifth loss by integrating the first loss calculated from the output of the region estimation model 14 and the fourth loss calculated from the output of the correspondence estimation model 124.

[0157] The optimizer 338 calculates the amount of parameter updates for each of the three models—the language feature extraction model 12, the region estimation model 14, and the correspondence estimation model 124—from the calculated fifth loss, and includes instructions to execute the process of updating the parameters of each model. The other configurations may be the same as those in Figure 8.

[0158] Figure 17 is a flowchart of the machine learning method performed by the machine learning device 70. In step S200, the processor 102 acquires a data set containing multiple observation sentences TXi, TXk, corresponding images IMi, IMk, and location information TPi, TPk related to regions of interest ROIi, ROIk in the images IMi, IMk (i ≠ k).

[0159] In step S210, the processor 102 inputs each observation sentence TXi and TXk into the language feature extraction model 12 and generates the respective observation features LFVi and LFVk.

[0160] In step S214, the processor 102 inputs each observation feature LFVi and LFVk into the correspondence estimation model 124 and estimates a correlation score that indicates the relationship between the two.

[0161] In step S220, the processor 102 inputs the combinations of each finding feature TXi, TXk and the images IMi, IMk into the region estimation model 14 to estimate the lesion region.

[0162] In step S226, the processor 102 uses the correlation score output from the correspondence estimation model 124 and correct answer Calculate the loss that shows the error with the relevance score.

[0163] In step S230, the processor 102 calculates a loss that represents the error between the location of the lesion region estimated by the region estimation model 14 and the location of the ground truth region of interest.

[0164] In step S240, the processor 102 calculates the parameter update amounts for each model—the correspondence estimation model 124, the language feature extraction model 12, and the region estimation model 14—so that the combined loss obtained by integrating the loss calculated in step S226 and the loss calculated in step S230 is minimized.

[0165] In step S254, the processor 102 updates the parameters of each model according to the parameter update amount calculated in step S240. The operations from steps S200 to S254 described above may be performed in mini-batch units.

[0166] After step S254, in step S260, the processor 102 determines whether or not to terminate the learning process. Step S260 may be the same process as step S160 in Figure 4.

[0167] If the result of step S260 is No, the processor 102 returns to step S200. If the result of step S260 is Yes, the processor 102 terminates the flowchart shown in the figure.

[0168] [Variation of the 7th embodiment] Figures 15 and 16 illustrate an example in which a text-based observation is input to the language feature extraction model 12. However, as described in the fourth embodiment (Figure 10), a configuration in which structured text (structured observation) is input to the language feature extraction model 12 is also possible.

[0169] 《Eighth Embodiment》 In the eighth embodiment, an example of an information processing device 300 that performs a process to determine the correspondence between observation sentences using a trained language feature extraction model 12E generated by the method of the seventh embodiment is described.

[0170] Figure 18 is a block diagram schematically showing the functional configuration of the information processing device 300 according to the eighth embodiment. The information processing device 300 includes a data acquisition unit 302, document structure analysis units 54A and 54B, language feature extractors 13A and 13B, a correspondence relationship estimater 125, and a judgment result output unit 306. The functions of each part of the information processing device 300 can be realized by a combination of computer hardware and software. The information processing device 300 may be configured as a computer system including one or more computers.

[0171] The data acquisition unit 302 acquires multiple combinations of observation sentences TXa and TXb to be compared. The text structure analysis unit 54A performs structural analysis of observation sentences TXa and generates structured data TSa. Similarly, the text structure analysis unit 54B performs structural analysis of observation sentences TXb and generates structured data TSb. In Figure 15, for the sake of explanation, two text structure analysis units 54A and 54B are shown, but these are the same (common) text structure analysis unit 54.

[0172] The language feature extractors 13A and 13B are processing units to which a trained model has been applied, which is a language feature extraction model 12 trained by the machine learning method described in the sixth embodiment. The two language feature extractors 13A and 13B shown in Figure 15 are the same (common) language feature extractor.

[0173] The language feature extractor 13A receives structured data TSa as input and generates the corresponding finding feature LFVa. Similarly, the language feature extractor 13B receives structured data TSb as input and generates the corresponding finding feature LFVb.

[0174] It is also possible to configure the language feature extractors 13A and 13B to receive observation texts TXa and TXb as input instead of structured data TSa and TSb, and generate corresponding observation features LFVa and LFVb. In this case, the text structure analysis units 54A and 54B may be omitted.

[0175] The correspondence relationship estimator 125 is , theThis is a processing unit to which a trained model has been trained using the machine learning method according to the 6th embodiment, which is used to train the correspondence estimation model 124. The correspondence estimator 125 accepts input of a combination of observation features LFVa and LFVb, and determines whether or not they are in a corresponding relationship.

[0176] The judgment result output unit 306 performs output processing of the correspondence determination result output from the correspondence relationship estimator 125. The judgment result output unit 306 may output the determination result regarding whether or not there is a correspondence between the two observation sentences, or it may use the determination result to generate a list of similar observation sentence candidates and output the similar observation sentence candidate list.

[0177] Figure 19 is a block diagram showing an example of the hardware configuration of the information processing device 300. The hardware configuration of the information processing device 300 may be the same as the example shown in Figure 13. Regarding the configuration shown in Figure 19, see Figure 13. composition Elements identical or similar to each other are given the same code, and redundant explanations are omitted.

[0178] The computer-readable medium 504 of the information processing device 300 stores multiple programs, including a data acquisition program 532, a text structure analysis program 534, a language feature extraction model 12E, a correspondence relationship estimation model 124E, and a similar observation sentence candidate list generation program 546. The data acquisition program 532 includes an instruction to execute a process to acquire the observation sentence to be processed. The data acquisition program 532 may acquire data from a database (not shown) in which past reports are stored, or it may accept data input via an input device 552.

[0179] The similar observation sentence candidate list generation program 546 includes instructions to perform a process that searches for similar observation sentences from a database (not shown) based on the output of the correspondence relationship estimation model 124E, and generates a similar observation sentence candidate list that includes the extracted similar observation sentences.

[0180] Furthermore, the computer-readable medium 504 of the information processing device 300 includes a report analysis information storage unit 548. The report analysis information storage unit 548 stores information of the analysis results, including structured data obtained by the document structure analysis program 534. Other configurations may be the same as in Figure 13.

[0181] 《Ninth Embodiment》 In the ninth embodiment, an example of an information processing device 400 that performs a similarity search of observation sentences using observation features generated using a trained language feature extraction model 12E is described.

[0182] Figure 20 is a block diagram schematically showing the functional configuration of the information processing device 400 according to the ninth embodiment. The information processing device 400 comprises a report receiving unit 402, a language feature extractor 13, a similarity search unit 404, and a similarity candidate output unit 406. The information processing device 400 may also include a database storage unit 650. The database storage unit 650 may be an external device that is communicatively connected to the information processing device 400.

[0183] The functions of each part of the information processing device 400 can be realized by a combination of computer hardware and software. The information processing device 400 may be configured as a computer system including one or more computers.

[0184] The database storage unit 650 stores a database containing multiple data sets, each linked to a report text FTXj and a report feature FFVj extracted from that report text FTXj.

[0185] In the information processing device 400 of the ninth embodiment, a feature vector (observation feature FFVj) is calculated in advance for each of the large number of observation sentences FTXj included in past reports using the language feature extractor 13, and the observation sentences FTXj and the observation features FFVj are linked and stored in a database.

[0186] Then, the observation text receiving unit 402 receives the observation text QTx to be searched for as input, and the language feature extractor 13 calculates the observation feature QFv. The similarity search unit 404 calculates the distance between the vectors of the observation feature QFv and each observation feature FFVj that have been calculated in advance, and extracts multiple candidates with close distances as similar observation text candidates.

[0187] The similarity candidate output unit 406 performs output processing to present similar observation sentence candidates extracted by the similarity search unit 404 to the user.

[0188] With this configuration, candidate observations similar to the observation QTx received from the observation receipt unit 402 are extracted from the database and presented to the user as a candidate list.

[0189] Regarding programs that operate computers: It is possible to record a program that enables a computer to implement some or all of the processing functions of each of the machine learning devices 10, 20, 30, 32, 70, 50, 60, 300, and 400 described in each of the above embodiments onto a computer-readable medium, such as an optical disk, magnetic disk, or semiconductor memory, which is a tangible, non-temporary information storage medium, and to provide the program through this information storage medium.

[0190] Also 、 Instead of providing programs by storing them on tangible, non-temporary computer-readable media, it is also possible to provide program signals as a download service using telecommunication lines such as the Internet.

[0191] Furthermore, some or all of the processing functions in each of the above-mentioned devices may be implemented using cloud computing, and they can also be provided as SaaS (Software as a Service).

[0192] Regarding the hardware configuration of each processing unit: The hardware structure of the processing units that perform various processes such as the loss calculation units 16, 26, 126, parameter update units 18, 28, 28A, 128, and document structure analysis unit 40 in the machine learning device 10 etc. described in each of the embodiments above, and the data acquisition units 52, 62, 302, document structure analysis unit 54, language feature extractor 13, image feature extractor 23, cross-modal feature integrator 25, correspondence relationship estimator 125, judgment result output units 56, 306, CAD unit 64, observation text reception unit 402, similarity search unit 404, and similarity candidate output unit 406 in the information processing device 50 etc., is, for example, various processors as shown below.

[0193] Various types of processors include general-purpose processors such as CPUs, GPUs, and FPGAs (Field Programmable Gate Arrays), which are programmed logic devices (PLDs) whose circuit configurations can be changed after manufacturing, and dedicated electrical circuits, which are processors with circuit configurations specifically designed to perform particular tasks, such as ASICs (Application Specific Integrated Circuits).

[0194] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as clients and servers. Secondly, a configuration where a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0195] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0196] Advantages of the embodiments of this disclosure According to each embodiment of the present disclosure described above, the following effects can be obtained.

[0197] [1] The linguistic feature extraction model 12 is trained to output a feature vector, which is a feature vector containing features of the location of the region of interest in the image mentioned by the input observation text or structured observation. The linguistic feature extraction model 12E generated by the method described in the embodiments of this disclosure can generate a feature vector embedded with features relating to the location of the region of interest in the image from the input text. The feature vector generated by the linguistic feature extraction model 12E can be used for various purposes, such as determining the degree of relevance between an image and an observation text, or searching for similar observation texts and presenting candidates for similar reports.

[0198] [2] According to the method described in the embodiments of the present disclosure, when training the language feature extraction model 12, it is not necessary to prepare ground truth features (ground truth feature vectors) that will be ground truth data for the output of the language feature extraction model 12. Instead, the relationship between the text and the location of the region of interest in the image can be learned using a data set consisting of an image IMj, location information TPj of the region of interest ROIj in the image IMj, and the text of an observation statement or structured observation that describes the region of interest ROIj in the image IMj.

[0199] [3] According to the methods described in the embodiments of this disclosure, a high-performance language feature extraction model 12E can be generated even when the amount of training data is relatively small.

[0200] Regarding the types of medical images: The technology disclosed herein is not limited to CT images, but can also apply to various medical images acquired by various medical devices (modalities), such as MR images acquired using MRI (Magnetic Resonance Imaging) devices, ultrasound images that project human body information, PET images acquired using positron emission tomography (PET) devices, and endoscopic images acquired using endoscopes. The images covered by the technology disclosed herein are not limited to three-dimensional images, but may also be two-dimensional images.

[0201] Other application examples In the embodiments described above, images and findings in medical imaging diagnosis were used as examples, but the scope of this disclosure is not limited to these examples and can be applied to various types of images and text concerning areas of interest within those images, regardless of the application. For example, the technology of this disclosure can also be applied to combinations of images of structures and text concerning defects within those images.

[0202] "others" This disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the technical idea of ​​this disclosure. [Explanation of Symbols]

[0203] 10 Machine Learning Devices 12, 12A, 12B, 12E Language Feature Extraction Models 13,13A,13B Language Feature Extractors 14. Region Estimation Models 16 Loss calculation section 18 Parameter update section 20 Machine Learning Devices 2 2, 22E Image Feature Extraction Model 23 Image Feature Extractor 24,24E Cross-modal feature integration model 25 Cross-modal feature integrator 26 Loss calculation section 28,28A Parameter update section 30,32 Machine Learning Devices 40 Sentence Structure Analysis Department 50 Information Processing Devices 52 Data Acquisition Unit 54,54A,54B Sentence structure analysis department 56. Output unit for judgment result 60 Information Processing Devices 62 Data Acquisition Unit 64 CAD Department 66 Data Storage Unit 70 Machine Learning Devices 102 processors 104 Computer-readable media 106 Communication Interface 108 Input / Output Interfaces 110 Bus 112 memory 114 storage 124,124E Correspondence Estimation Model 125 Correspondence Estimator 126 Loss calculation section 128 Parameter update section 130 Learning Processing Program 132 Data Acquisition Program 136 Loss Calculation Program 138 Optimizer 140 Display Control Program 152 Input device 154 Display device 230 Learning Processing Program 232 Data Acquisition Program 236 Loss Calculation Program 238 Optimizer 300 Information Processing Devices 302 Data Acquisition Unit 304 Computer-readable media 306 Judgment Result Output Unit 330 Learning Processing Programs 332 Data Acquisition Program 336 Loss Calculation Program 338 Optimizer 400 Information Processing Devices 402 Section for Receiving Reports 404 Similarity Search Section 406 Similar Candidate Output Unit 502 Processors 504 Computer-readable media 506 Communication Interface 508 Input / Output Interfaces 510 Bus 512 memory 514 storage 532 Data Acquisition Program 534 Text Structure Analysis Program 536 Discrimination Result Presentation Program 538 Analysis information storage area 540 Display control program 546 Program for generating a list of candidate similar findings. 548 Findings analysis information storage unit 552 Input device 554 Display device 600 Training data storage unit 610 Medical Image Storage Department 612 Report Storage Section 650 Database Storage Unit TDj training data IMi, IMj, IMk, IMx images ROIi,ROIj,ROIk,ROIx Region of interest TXi,TXj,TXk,TXy,TXa,TXb Observations LFVj, LFVy, LFVa, LFVb findings and characteristics IFVj , IFVx Image Features TPi, TPj, TPk, TPx location information PAj estimated area information TSj, TSY, TSa, TSb structured data FTXj Observations FFVj findings and characteristics QTx findings QFv findings and characteristics S100~S160 Steps in the Machine Learning Method S200~S260 Steps in the Machine Learning Method

Claims

1. A method for generating a language feature extraction model that causes a computer to perform a process of extracting features from text associated with an image, A system containing one or more processors Machine learning is performed using multiple training data sets, each containing a first image, first positional information relating to a region of interest in the first image, and first text describing the region of interest. The first text is input to the first model and the first model outputs a first feature quantity that represents the characteristics of the first text. The first image and the first feature quantities are input to a second model, which is different from the first model, and the second model is made to estimate the region of interest in the first image. By training the first and second models such that the estimated region of interest output from the second model matches the correct region of interest indicated by the first position information, The first model, which is the language feature extraction model, is generated. Method for generating language feature extraction models.

2. The aforementioned system A third model is used that receives image features extracted from the aforementioned image and language features extracted from the aforementioned text as input and outputs the degree of association between the two. In the aforementioned machine learning, the third model is input with the second feature extracted from the first image and the first feature to estimate the degree of association between the first image and the first text. This includes training the first and third models so that the estimated relevance output from the third model matches the ground truth relevance. A method for generating a language feature extraction model according to claim 1.

3. The aforementioned system Using a fourth model that extracts the second feature quantity from the input first image, In the aforementioned machine learning, The first image and the position information are input to the fourth model, and the fourth model outputs the second feature quantity. This includes training the first model, the third model, and the fourth model such that the estimated relevance output from the third model matches the relevance of the ground truth. A method for generating a language feature extraction model according to claim 2.

4. The aforementioned system A fifth model is used that receives input of linguistic features extracted from each of the multiple texts and outputs the degree of relevance of the multiple texts, In the aforementioned machine learning, By inputting a second text, different from the first text, into the first model, the third feature quantity extracted from the second text by the first model, along with the first feature quantity, is input into the fifth model to allow the fifth model to estimate the degree of association between the first text and the second text. This includes training the first model and the fifth model so that the estimated relevance output from the fifth model matches the ground truth relevance. A method for generating a language feature extraction model according to claim 1.

5. The above text and the first text are structured texts. A method for generating a language feature extraction model according to any one of claims 1 to 4.

6. The aforementioned second text is structured text. A method for generating a language feature extraction model according to claim 4.

7. The aforementioned system This includes a process that displays the region of interest estimated by the second model described above. A method for generating a language feature extraction model according to claim 1.

8. The position information includes coordinate information that identifies the location of the region of interest in the first image, A method for generating a language feature extraction model according to claim 1.

9. The first image is a cropped image containing the position information. A method for generating a language feature extraction model according to claim 1.

10. One or more storage devices that store a program including the language feature model generated by the method for generating a language feature model according to claim 1, One or more processors that execute the aforementioned program, An information processing device equipped with the following features.

11. One or more processors, The system comprises one or more storage devices that store instructions executed by the one or more processors, The one or more processors described above are: Get the text that describes the region of interest in the image. The process involves inputting the text into the first model and outputting linguistic features representing the characteristics of the text from the first model. The first model described above is Machine learning using multiple training data sets, including a first training image, first location information relating to a region of interest in the first image, and first text describing the region of interest, The first model is given the first text as input and the first model outputs a first feature quantity that represents the features of the first text; the first image and the first feature quantity are given to a second model different from the first model and the second model is made to estimate the region of interest in the first image; This model is obtained by training the first and second models such that the estimated region of interest output from the second model matches the correct region of interest indicated by the first position information. Information processing device.

12. The one or more processors described above are: Image features extracted from the second image and language features extracted from the text are input to a third model, and the third model outputs the degree of association between the second image and the text. The information processing apparatus according to claim 10 or 11.

13. The one or more processors described above are: The second image and the second position information relating to the region of interest in the second image are obtained. By inputting the second image and the second location information into the fourth model, the fourth model is made to output the image features. The information processing apparatus according to claim 12.

14. The one or more processors described above are: The language features extracted from each of the multiple texts by the first model are input to the fifth model, and the relevance of the multiple texts is output from the fifth model. The information processing apparatus according to claim 10 or 11.

15. The above text and the first text are structured texts. The information processing apparatus according to claim 10 or 11.

16. One or more processors, Get the text that describes the region of interest in the image. The process involves inputting the text into the first model and outputting linguistic features representing the characteristics of the text from the first model. The first model described above is Machine learning using training data including a first image for training, a first text describing a region of interest in the first image, and first positional information relating to the region of interest in the first image, The first model is given the first text as input and the first model outputs a first feature quantity that represents the features of the first text; the first image and the first feature quantity are given to a second model different from the first model and the second model is made to estimate the region of interest in the first image; This model is obtained by training the first model and the second model such that the region of interest estimated by the second model coincides with the region of interest indicated by the first location information. Information processing methods.

17. A program that enables a computer to perform the function of extracting features from text associated with an image, To the aforementioned computer, A function to retrieve text describing the region of interest in the image, The system implements a function that inputs the text into the first model and outputs linguistic features representing the characteristics of the text from the first model, The first model described above is Machine learning using training data including a first image for training, first positional information relating to a region of interest in the first image, and first text describing the region of interest in the first image, The first model is given the first text as input and the first model outputs a first feature quantity that represents the features of the first text; the first image and the first feature quantity are given to a second model different from the first model and the second model is made to estimate the region of interest in the first image; This model is obtained by training the first and second models such that the estimated region of interest output from the second model matches the region of interest indicated by the first position information. program.

Images