Methods for generating pre-trained models, machine learning systems, and programs

Abstract

Provided are a learned model generation method, a machine learning system, a program, and a medical image processing device, that make it possible to implement domain conversion learning that is robust in terms of misalignment of data to be used for learning. A computer acquires a plurality of learning data items including a first-domain medical image and a second-domain medical image, and performs learning processing by using a learning model having the structure of a generative adversarial network that includes: a first generator configured using a first convolutional neural network that receives an input of a medical image of a first domain and that outputs a first generated image of a second domain; and a first discriminator configured using a second convolutional neural network that receives an input of data including first image data, which is a second-domain medical image contained in the first generated image or a data set for learning, and coordinate information of a human coordinate system corresponding to the respective positions of a plurality of unit elements constituting the first image data, and that discriminates the authenticity of an inputted image.

Description

Technical Field

[0001] The present invention relates to a method for generating a learned model, a machine learning system, a program, and a medical image processing apparatus, and particularly relates to machine learning technology and image processing technology for handling medical images.

Background Art

[0002] In the medical field, image diagnosis is performed using medical images taken by various modalities such as a CT (Computed Tomography) apparatus or an MRI (Magnetic Resonance Imaging) apparatus. In recent years, the development of artificial intelligence (AI) that extracts parts such as organs, detects lesion regions, or classifies disease names from medical images using deep learning has been promoted.

[0003] Patent Document 1 describes a diagnostic support system that extracts an organ region from a medical image using AI. Patent Document 2 describes a machine learning method for obtaining a learning model that generates an MR estimated image obtained by estimating an MR (Magnetic Resonance) image from a CT image.

[0004] Non-Patent Document 1 describes a method for generating a T2-weighted image of MRI from a CT image using machine learning. Non-Patent Document 2 proposes a method of adding a channel representing the coordinate information of each pixel in an image and incorporating position information into a convolutional neural network.

[0005] Non-Patent Document 3 discloses a technique that can learn the mutual conversion of heterogeneous domain images using a network that combines two configurations of a generative adversarial network (GAN) and using a dataset for each domain without using paired images as learning data.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-149094 [Patent Document 2] Japanese Patent Publication No. 2020-54579 [Non-patent literature]

[0007] [Non-Patent Document 1] Cheng-Bin Jin, Hakil Kim, Mingjie Liu, Wonmo Jung, Seongu Joo, Eunsik Park, Young Saem Ahn, In Ho Han, Jae Il Lee, Xuenan Cui, "Deep CT to MR Synthesis Using Paired and Unpaired Data", Sensors 2019.19(10), 2361 [Non-Patent Document 2] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, Jason Yosinski, "An intriguing failing of convolutional neural networks and the CoordConv solution", ArXiv: 1807.03247 [Non-Patent Document 3] Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", ArXiv: 1703.10593 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Medical images are generated using various modalities, and the characteristics of the images differ depending on the modality. Computer-aided diagnosis (CAD) systems using AI are generally built for each modality used to capture the target medical image. If technologies developed for a specific modality can be adapted to images from other modalities, their use in a wider range of situations can be expected.

[0009] For example, if a system for organ extraction CAD has been built that uses CT images as input to extract organ regions, it becomes possible to apply this technology to similarly extract organ regions from MR (Magnetic Resonance) images.

[0010] To achieve this, a high-performance image converter is required that can perform image conversion between different modalities, such as a process to generate pseudo-MR images from CT images, or conversely, a process to generate pseudo-CT images from MR images. Note that "image conversion" can be rephrased as "image generation," and "converter" can be rephrased as "generator."

[0011] A typical method for training a deep learning-based algorithm to perform such heterogeneous image transformation tasks is CycleGAN, described in Non-Patent Document 3. CycleGAN prepares datasets belonging to two different domains and learns the interconversion between these domains. The features of the generated images produced by the trained model depend on the data used for training. Therefore, for example, when training a CycleGAN model using a dataset belonging to the CT domain and a dataset belonging to the MR domain, it is assumed that these datasets represent images of the same body region. If there is a large positional shift between the datasets, there will be regions that can only be observed in the data from one of the domains. When training with data that has such a large positional shift, the model will not be able to correspond between images from different domains, and proper training will not occur.

[0012] The technology described in Non-Patent Document 1 is sensitive to positional misalignment between datasets, and if data with large positional misalignments is used for training, the generated images may become corrupted.

[0013] The challenges described above are not limited to CycleGAN, but are common challenges in training image transformation tasks using GAN algorithms.

[0014] This disclosure is made in view of these circumstances and aims to provide a method for generating a trained model, a machine learning system, a program, and a medical image processing device that can achieve robust transformation learning against positional shifts between datasets used for training. [Means for solving the problem]

[0015] A method for generating a trained model according to one aspect of this disclosure is a method for generating a trained model that transforms the domain of an input medical image and outputs a generated image of a different domain, and includes a learning model having the structure of a generative adversarial network, which includes: a first generator configured using a first convolutional neural network that accepts input medical images of a first domain and outputs a first generated image of a second domain different from the first domain; and a first discriminator configured using a second convolutional neural network that accepts input data including a first generated image generated by the first generator or a medical image of a second domain included in a training dataset, and coordinate information of a human body coordinate system corresponding to the positions of each of a plurality of unit elements constituting the first image data, and determines the authenticity of the input image, wherein the computer acquires a plurality of training data including medical images of a first domain and medical images of a second domain, and performs a learning process to train the first generator and the first discriminator adversarially based on the plurality of training data.

[0016] According to this embodiment, coordinate information of the human body coordinate system is introduced to the medical image used for training, and the first discriminator is given data including the first image data, which is the image to be judged as true or false, and coordinate information corresponding to each of the multiple unit elements in the first image data. The first discriminator learns true or false according to the position indicated by the coordinate information by performing convolution on this data. According to this embodiment, robustness to positional shifts in the data used for training is improved, and appropriate image transformation (image generation) learning can be realized. The unit elements of a 3D image may be understood as voxels, and the unit elements of a 2D image may be understood as pixels.

[0017] In a method for generating a trained model according to another aspect of this disclosure, the coordinate information corresponding to the first generated image when the first generated image is input to the first discriminator may be the coordinate information defined for the medical image of the first domain, which is the source image input to the first generator when generating the first generated image.

[0018] In a method for generating a trained model according to another aspect of this disclosure, the first image data may be three-dimensional data, and the coordinate information may include x-coordinate information, y-coordinate information, and z-coordinate information that specify the position of each voxel, which is a unit element, in three-dimensional space, and the x-coordinate information, y-coordinate information, and z-coordinate information may be combined as channels into channels of the first image data or a feature map of the first image data and provided to the first discriminator.

[0019] In a method for generating a trained model according to another aspect of this disclosure, the coordinate information of the human body coordinate system may be absolute coordinates defined based on the anatomical position of a part of the human body, and the configuration may be such that coordinate information corresponding to each unit element in the image is associated with each medical image used as training data.

[0020] In a method for generating a trained model according to another aspect of this disclosure, the computer may be configured to generate coordinate information corresponding to each unit element in each medical image used as training data.

[0021] In the method for generating a learned model according to another aspect of the present disclosure, the configuration may be such that coordinate information is input in the intermediate layer of the second convolutional neural network.

[0022] In the method for generating a learned model according to another aspect of the present disclosure, the learning model further includes a second generator configured using a third convolutional neural network that receives an input of a medical image of a second domain and outputs a second generated image of a first domain, second image data that is a medical image of the first domain included in the second generated image generated by the second generator or the learning dataset, and coordinate information of a human body coordinate system corresponding to each position of a plurality of unit elements constituting the second image data, and receives an input of data including the same, and a second discriminator configured using a fourth convolutional neural network that discriminates the authenticity of the input image. The learning process may include a process of adversarially training the second generator and the second discriminator.

[0023] In the method for generating a learned model according to another aspect of the present disclosure, the coordinate information corresponding to the second generated image when the second generated image is input to the second discriminator may be coordinate information defined for the medical image of the second domain that is the source image input to the second generator when generating the second generated image.

[0024] In the method for generating a learned model according to another aspect of the present disclosure, the computer calculates a first reconstruction loss of a conversion process using the first generator and the second generator in this order based on a first reconstructed generated image output from the second generator by inputting the first generated image of the second domain output from the first generator to the second generator, and calculates a second reconstruction loss of a conversion process using the second generator and the first generator in this order based on a second reconstructed generated image output from the first generator by inputting the second generated image of the first domain output from the second generator to the first generator. The configuration may include performing the above processes.

[0025] In a method for generating a trained model described in another aspect of this disclosure, the medical image in the first domain is a first modality image taken using a first modality which is a medical device, and the medical image in the second domain is a second modality image taken using a second modality which is a different type of medical device from the first modality, and the trained model may be configured to receive the input of the first modality image and be trained to generate a pseudo-second modality generated image that has the features of an image taken using the second modality.

[0026] Other aspects of the present disclosure relate to a machine learning system for training a learning model that transforms the domain of an input medical image to generate a generated image of a different domain, comprising at least one first processor and at least one first memory device that stores a program executed by the at least one first processor, wherein the learning model has the structure of an adversarial generative network, comprising: a first generator configured using a first convolutional neural network that accepts input medical images of a first domain and outputs a first generated image of a second domain different from the first domain; and a first discriminator configured using a second convolutional neural network that accepts input data including first image data which is the first generated image generated by the first generator or a medical image of a second domain included in a training dataset, and coordinate information of a human body coordinate system corresponding to the positions of each of a plurality of unit elements constituting the first image data, and determines the authenticity of the input image, wherein at least one first processor acquires a plurality of training data including medical images of a first domain and medical images of a second domain by executing instructions of the program, and performs a learning process to train the first generator and the first discriminator adversarially based on the plurality of training data.

[0027] A program relating to another aspect of this disclosure is a program that causes a computer to perform a process to train a learning model that transforms the domain of an input medical image to generate a generated image of a different domain, wherein the learning model has the structure of an adversarial generative network including: a first generator configured using a first convolutional neural network that accepts input medical images of a first domain and outputs a first generated image of a second domain different from the first domain; and a first discriminator configured using a second convolutional neural network that accepts input data including first image data which is the first generated image generated by the first generator or a medical image of a second domain included in a training dataset, and coordinate information of a human body coordinate system corresponding to the positions of each of a plurality of unit elements constituting the first image data, and determines the authenticity of the input image, thereby causing the computer to acquire a plurality of training data including medical images of a first domain and medical images of a second domain, and to perform a learning process that trains the first generator and the first discriminator adversarially based on the plurality of training data.

[0028] A medical image processing device according to another aspect of the present disclosure comprises a second storage device that stores a first trained model, which is a trained first generator trained by implementing a method for generating a trained model according to any one aspect of the present disclosure, and a second processor that performs image processing using the first trained model, wherein the first trained model is a model that receives a first medical image as input and outputs a second medical image of a different domain from the first medical image. [Effects of the Invention]

[0029] According to the present invention, robustness to positional shifts in the data used for training is improved, and even when using image data with positional shifts, it is possible to learn appropriate domain transformations. According to the present invention, it is possible to obtain a trained model that outputs appropriate generated images of different domains for input medical images. Furthermore, by using the trained model generated by the present invention, it is possible to obtain high-quality pseudo-images (generated images) that have features of different domains. [Brief explanation of the drawing]

[0030] [Figure 1] Figure 1 is an explanatory diagram illustrating the challenges in modality conversion of medical images. [Figure 2] Figure 2 shows examples of MR images included in an MR dataset and CT images included in a CT dataset. [Figure 3] Figure 3 shows an example of an image converted from MR to CT. [Figure 4] Figure 4 is a conceptual diagram showing an overview of the processing in the machine learning system according to the first embodiment. [Figure 5] Figure 5 is an explanatory diagram of the human body coordinate system applied to the first embodiment. [Figure 6] Figure 6 shows an example of coordinate information added to an image. [Figure 7] Figure 7 is a functional block diagram showing an example configuration of a machine learning system according to the first embodiment. [Figure 8] Figure 8 is a functional block diagram showing an example of the configuration of the training data generation unit. [Figure 9] Figure 9 shows an example of a pseudo-MR image generated by a trained model trained by a learning process using the machine learning system according to the first embodiment. [Figure 10] Figure 10 is a functional block diagram showing an example configuration of a machine learning system according to the second embodiment. [Figure 11] Figure 11 is a schematic diagram showing the processing flow when CT input occurs in the machine learning system according to the second embodiment. [Figure 12] Figure 12 is a schematic diagram showing the processing flow when MR input is received in the machine learning system according to the second embodiment. [Figure 13] Figure 13 is a block diagram showing an example configuration of an information processing device applied to a machine learning system. [Figure 14] Figure 14 is a block diagram showing an example configuration of a medical image processing device to which a trained model generated by performing a training process using a machine learning system is applied. [Figure 15] Figure 15 is a block diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0031] Preferred embodiments of the present invention will be described below with reference to the attached drawings.

[0032] [Explanation of medical images] Typical examples of devices used to acquire medical images include CT scanners and MRI scanners. In these modalities, the basic idea is to obtain three-dimensional data representing the three-dimensional form of an object by sequentially acquiring two-dimensional slice images. In this specification, the term "three-dimensional data" includes the concept of a collection of sequentially acquired two-dimensional slice images and is synonymous with "three-dimensional image." The term "image" includes the meaning of image data. A collection of consecutive two-dimensional slice images is sometimes called a "two-dimensional image sequence" or "two-dimensional image series." The term "two-dimensional image" includes the concept of a two-dimensional slice image extracted from three-dimensional data.

[0033] Challenges in Modality Conversion of Medical Images Figure 1 is an explanatory diagram illustrating the challenges in modality conversion of medical images. Here, we show an example of training a model to learn mutual conversions, such as converting from CT images to MR images and from MR images to CT images, using CT images and MR images as training data. Note that both CT and MR images are 3D data.

[0034] When training a model to perform CT-MR conversion using dataset A, which contains multiple CT images, and dataset B, which contains multiple MR images, there may be discrepancies in the image positions between datasets, as shown in the figure. It is difficult to directly train a model to perform modality conversion using datasets with such positional discrepancies. Note that the term "positional discrepancy" includes both differences in the position of the imaging area and differences in the size of the imaging area. For example, in the example shown in Figure 1, the imaging area of ​​the MR images in dataset B is larger than that of the CT images in dataset A, so there are areas that are visible in the MR images but not in the CT images.

[0035] Figures 2 and 3 show specific examples. Figure 2 shows an example of an MR image included in an MR dataset and a CT image included in a CT dataset. As shown in the figure, although the MR image and the CT image have a partial overlap in their imaging areas, there is a difference in their imaging areas, and the MR image captures a wider area than the CT image.

[0036] Figure 3 shows an example of generated images when the Generative Adversarial Network (GAN) of the comparative example is trained using a dataset with positional misalignment between domains. The GAN of the comparative example has a configuration that extends the network structure described in Non-Patent Document 3 to three dimensions.

[0037] Figure 3 shows an example of an MR to CT conversion image, with the original MR image on the left and the converted CT image on the right. As shown in Figure 3, the converted CT image directly reflects the positional shift of the dataset used for training.

[0038] 《First Embodiment》 Figure 4 is a conceptual diagram showing an overview of the processing in the machine learning system 10 according to the first embodiment. Here, we will describe a method for learning an image transformation task that generates a pseudo-MR image from a CT image, with the source domain being CT and the target domain being MR, based on a GAN architecture.

[0039] The machine learning system 10 includes a generator 20G and a discriminator 24D. Both the generator 20G and the discriminator 24D are constructed using a 3D convolutional neural network (CNN). The generator 20G is a 3D generative network (3D generator) that receives 3D data with features of the CT domain as input and outputs 3D data with features of the MR domain. For example, the generator 20G is fitted with a V-net type architecture, which is a 3D extension of U-net.

[0040] U-Net is a neural network widely used for medical image segmentation and other applications. For example, a document describing U-Net is "Olaf Ronneberger, et al. "U-Net: Convolutional Networks for Biomedical Image Segmentation", MICCAI, 2015". Similarly, a document describing V-Net is "Fausto Milletari, et al. "V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation"".

[0041] The discriminator 24D is a three-dimensional discriminator network (3D discriminator) that determines the authenticity of an image. In the machine learning system 10 according to the first embodiment, human body coordinate system coordinate information is added to the images used for training, and coordinate data indicating the human body coordinate system coordinate information corresponding to the image region is added to the input data for the discriminator 24D. The coordinate information includes x-coordinate information, y-coordinate information, and z-coordinate information that identify the position of each voxel constituting the image in three-dimensional space.

[0042] In other words, in the machine learning system 10, three channels (3ch) of coordinate data (x, y, and z coordinates) are added to the input data for the classifier 24D, and 4ch data, which is a combination of the image channel (1ch) and the coordinate channel (3ch), is input to the classifier 24D.

[0043] The discriminator 24D is input with data including generated images, which are pseudo-MR images generated by the generator 20G, or image data of real MR images included in the training dataset, and corresponding coordinate information. The discriminator 24D then determines whether the image is a real image or a fake image generated by the generator 20G. The image data input to the discriminator 24D is an example of the "first image data" in this disclosure.

[0044] "Real image" refers to an actual image obtained by actually taking a photograph using a photographic device. "Fake image" refers to a generated image (pseudo-image) that is artificially produced by image conversion processing without taking a photograph. In the first embodiment, the data used as training data to be input to the learning model 44 is a "real image," and the generated image produced by the generator 20G is a "fake image."

[0045] Figure 5 is an explanatory diagram of the human body coordinate system applied to the first embodiment. In this specification, the body axis direction is defined as the z-axis direction, the horizontal direction (left-right direction) in a standing human body is defined as the x-axis direction, and the depth direction (front-back direction) is defined as the y-axis direction. In the human body coordinate system, the z-coordinate is defined as "-1.0" for the top of the head and "1.0" for the toes. The x-coordinate and y-coordinate are also defined as "-1.0 to 1.0" in the range that encompasses the entire human body, similar to the z-coordinate.

[0046] The definition of a human body coordinate system is not limited to this example; it is sufficient to define a coordinate system that can specify spatial positions as absolute coordinates based on the anatomical positions of parts of the human body. In other words, a human body coordinate system is an absolute coordinate system defined based on the anatomical positions of parts of the human body, and the coordinate values ​​of a human body coordinate system have meaning as absolute coordinate values ​​even across different images.

[0047] The data used for training can be generated, for example, by extracting a portion from an image of the patient's entire body (whole-body image). If a whole-body image is available, the x, y, and z coordinates can be defined according to the definition above, and coordinate information can be associated with each voxel. In addition to whole-body images, if partial images such as the upper body, chest, or pelvis are used, the values ​​of the x, y, and z coordinates can be determined by identifying anatomical landmarks within the image and comparing them with a standard human anatomy atlas.

[0048] When cropping the image region data used for training from the original 3D data, the coordinate information is also cropped, thereby associating (linking) the cropped 3D data with its corresponding coordinate information. The image region to be cropped can be determined randomly.

[0049] Figure 6 shows an example of coordinate information added to an image. Here, the image channel is denoted as ch1, the z-coordinate information channel as ch2, the y-coordinate information channel as ch3, and the x-coordinate information channel as ch4. The coordinate information for each axis can be treated as image data by representing the coordinate values ​​with grayscale. Each coordinate channel from ch2 to ch4 can become data corresponding to a grayscale image that reflects the coordinate values.

[0050] 《Example Configuration of Machine Learning System 10》 Figure 7 is a functional block diagram showing an example configuration of the machine learning system 10 according to the first embodiment. The machine learning system 10 includes a training data generation unit 30 and a training processing unit 40. The machine learning system 10 may further include an image storage unit 50 and a training data storage unit 54.

[0051] The machine learning system 10 can be implemented by a computer system including one or more computers. The functions of the training data generation unit 30, the training processing unit 40, the image storage unit 50, and the training data storage unit 54 can be implemented by a combination of computer hardware and software. The functions of each of these units may be implemented by a single computer, or the processing functions may be shared among two or more computers.

[0052] Here, we describe an example in which the learning data generation unit 30, the learning processing unit 40, the image storage unit 50, and the learning data storage unit 54 are each configured as separate devices. For example, the learning data generation unit 30, the learning processing unit 40, the image storage unit 50, and the learning data storage unit 54 may be connected to each other via a telecommunications line. The term "connection" is not limited to wired connections, but also includes the concept of wireless connections. The telecommunications line may be a local area network or a wide area network. With this configuration, the learning data generation process and the generation model learning process can be carried out without being physically or temporally constrained by each other.

[0053] The image storage unit 50 includes a large-capacity storage device for storing CT reconstructed images (CT images) taken by a medical X-ray CT scanner and MR reconstructed images (MR images) taken by an MRI scanner. The image storage unit 50 may be, for example, a DICOM server that stores medical images in accordance with the DICOM (Digital Imaging and Communications in Medicine) standard. The medical images stored in the image storage unit 50 may be images of individual parts of the human body or images of the entire body.

[0054] The learning data generation unit 30 generates training data (training data) to be used for machine learning. "Training data" is synonymous with "training data". In the machine learning system 10, the learning data consists of a dataset containing multiple 3D data sets that are actual CT images taken using a CT scanner, and a dataset containing multiple 3D data sets that are actual MR images taken using an MRI scanner. Each 3D data set is associated with coordinate information for each voxel. Such learning data can be generated from data stored in the image storage unit 50. A voxel is an example of a "unit element" in this disclosure.

[0055] The training data generation unit 30 acquires the original 3D data from the image storage unit 50, performs preprocessing such as generating coordinate information and cropping a fixed-size region, and generates 3D data with coordinate information of a desired image size suitable for input to the training processing unit 40. In order to efficiently carry out the training process by the training processing unit 40, multiple training data sets may be generated in advance using the training data generation unit 30 and stored in storage as training datasets.

[0056] The learning data storage unit 54 includes storage for storing pre-processed learning data generated by the learning data generation unit 30. The learning data generated by the learning data generation unit 30 is read from the learning data storage unit 54 and input to the learning processing unit 40.

[0057] The learning data storage unit 54 may be included in the learning data generation unit 30, or a portion of the memory area of ​​the image storage unit 50 may be used as the learning data storage unit 54. In addition, some or all of the processing functions of the learning data generation unit 30 may be included in the learning processing unit 40.

[0058] The learning processing unit 40 includes a data acquisition unit 42 and a learning model 44 having a GAN structure. The data acquisition unit 42 acquires learning data to be input to the learning model 44 from the learning data storage unit 54. The learning data acquired via the data acquisition unit 42 is input to the learning model 44. The learning model 44 is composed of a generator 20G and a discriminator 24D. The learning processing unit 40 also includes a coordinate information merging unit 22 that combines coordinate information with the generated image output from the generator 20G. The coordinate information merging unit 22 combines the coordinate information associated with the input image that served as the source (transformation source) of the generated image with the generated image and provides it to the discriminator 24D.

[0059] The learning processing unit 40 further includes an error calculation unit 46 and an optimizer 48. The error calculation unit 46 evaluates the error between the output from the discriminator 24D and the correct answer using a loss function. Error can also be referred to as loss.

[0060] The optimizer 48 updates the network parameters in the learning model 44 based on the calculation results of the error calculation unit 46. The network parameters include the filter coefficients (weights of connections between nodes) and node biases of the filters used in processing each layer of the CNN.

[0061] In other words, the optimizer 48 performs a parameter calculation process to calculate the amount of parameter updates for the generator 20G and the discriminator 24D networks based on the calculation results of the error calculation unit 46, and a parameter update process to update the parameters for the generator 20G and the discriminator 24D networks according to the calculation results of the parameter calculation process. The optimizer 48 updates the parameters based on algorithms such as gradient descent.

[0062] The learning processing unit 40 trains the learning model 44 to improve the performance of each network by repeatedly performing adversarial learning between the generator 20G and the discriminator 24D based on the input learning data.

[0063] Regarding the generation of training data: Figure 8 is a functional block diagram showing an example configuration of the learning data generation unit 30. The learning data generation unit 30 includes a coordinate information generation unit 33 and a cropping processing unit 34. The coordinate information generation unit 33 performs a process to generate coordinate information in the human body coordinate system for the position of each voxel in the original 3D data (original 3D image) to be processed. The coordinate information generation unit 33 assigns coordinate values ​​in the human body coordinate system to each voxel in the original 3D image according to the definition of the human body coordinate system explained in Figure 5.

[0064] The cropping processing unit 34 randomly extracts a fixed-size region from the original 3D image, which is associated with coordinate information. When cropping the image region, the cropping processing unit 34 also crops the coordinate information. The 3D data extracted into a fixed-size region by the cropping processing unit 34 is linked to the coordinate information and stored in the learning data storage unit 54.

[0065] The original 3D data input to the training data generation unit 30 may be a CT image or an MR image. Furthermore, the training data may be understood as either cropped, fixed-size 3D data or the original 3D data before cropping.

[0066] Learning method in the first embodiment In the first embodiment, the data used for training may be a domain-specific dataset, as explained in Figure 1, and data may be randomly extracted from each domain's dataset. It should be noted that the machine learning system 10 according to the first embodiment does not exclude the possibility of training using paired images; for example, training can be performed using paired images of the same shooting area captured with different modalities as training data.

[0067] In the machine learning system 10 of the first embodiment, when image data is input to the discriminator 24D, coordinate data corresponding to that image data is input. When a generated image (pseudo-image) generated by the generator 20G is input to the discriminator 24D, the coordinate data corresponding to this generated image is the coordinate data defined for the source image input to the generator 20G. On the other hand, when a real image included in the training dataset is input to the discriminator 24D, the coordinate data associated with this real image is input to the discriminator 24D.

[0068] The discriminator 24D performs convolution on the input image data and coordinate data to determine whether the data is true or false. Adversarial learning is performed on the generator 20G and the discriminator 24D using a GAN algorithm, and the discriminator 24D is trained to determine whether the data is true or false according to the position indicated by the coordinate information. According to the first embodiment, image transformation that is robust to positional shifts between datasets can be achieved.

[0069] The method for generating a trained generator 20G by a learning process using the machine learning system 10 is an example of a "method for generating a trained model" in this disclosure. Generator 20G is an example of a "first generator" in this disclosure, and the 3D CNN used in generator 20G is an example of a "first convolutional neural network" in this disclosure. Discriminator 24D is an example of a "first generator" in this disclosure, and the 3D CNN used in discriminator 24D is an example of a "second convolutional neural network" in this disclosure. The CT domain is an example of a "first domain" in this disclosure, and the MR domain is an example of a "second domain" in this disclosure. The CT image input to generator 20G is an example of a "medical image of the first domain" and a "first modality image" in this disclosure. The pseudo-MR image generated by generator 20G is an example of a "first generated image" in this disclosure. The pseudo-MR image output from generator 20G is an example of a "second modality generated image" in this disclosure. The CT scanner and MRI scanner are examples of “medical devices” in this disclosure. The CT scanner is an example of “first modality” in this disclosure, and the MRI scanner is an example of “second modality” in this disclosure. The MR image, which is the actual image input to the discriminator 24D, is an example of “second domain medical image” and “second modality image” in this disclosure.

[0070] Figure 9 shows an example of a pseudo-MR image generated by a trained model trained using the machine learning system 10 according to the first embodiment. The left side shows the source CT image, and the right side shows the converted pseudo-MR image. The converted pseudo-MR image output from the trained model is an image of the same area as the input CT image. As a result of training according to the first embodiment, the trained model can appropriately generate pseudo-MR images without positional shift by performing domain transformation from CT images, as shown in Figure 9.

[0071] 《Example 1》 In the first embodiment, an example was shown in which four channels, consisting of an image channel and a coordinate channel, are input to the input layer of the discriminator 24D. However, the coordinate information may be input to any of the intermediate layers in the CNN that constitutes the discriminator 24D. In that case, the coordinate data is processed by pooling or other methods on the original coordinate data to match the number of voxels of the feature map of the image data, and then the coordinate channels are combined with the channels of the feature map before being provided to the discriminator 24D.

[0072] 《Modified Example 2》 In the first embodiment, an example using a 3D CNN for 3D images was described, but it is also possible to apply a 2D CNN for 2D images. In the case of 2D images, the definition of the human body coordinate system is the same as in the case of 3D images, and the coordinate information for a 2D image may be 2D coordinate data corresponding to each pixel that makes up the image.

[0073] 《Second Embodiment》 In the second embodiment, we will describe an example in which an architecture based on the CycleGAN mechanism described in Non-Patent Document 2 is adopted, and images from each domain that are not related (not paired) are used as training data to learn the domain transformation task.

[0074] Figure 10 is a functional block diagram showing an example configuration of the machine learning system 210 according to the second embodiment. In Figure 10, elements that are the same as or similar to those in Figure 6 are denoted by the same reference numerals, and redundant explanations are omitted.

[0075] The learning data storage unit 54 shown in Figure 10 stores the original 3D data belonging to the respective domains of CT and MR.

[0076] The machine learning system 210 includes a learning processing unit 240 instead of the learning processing unit 40 in Figure 6. The learning processing unit 240 includes a data acquisition unit 42, a preprocessing unit 230, a learning model 244, an error calculation unit 246, and an optimizer 248.

[0077] The preprocessing unit 230 performs the same processing as the learning data generation unit 30 described in Figure 8, and includes a coordinate information generation unit 33 and a cropping unit 34. The preprocessing unit 230 performs preprocessing on the 3D data acquired via the data acquisition unit 42 for input to the learning model 244. Here, coordinate information generation processing and cropping processing are given as examples of preprocessing, but these processes can be performed as needed, and some or all of the processing in the preprocessing unit 230 can be omitted.

[0078] For example, as explained in Figure 8, preprocessing may be performed in advance, and the preprocessed dataset may be stored in the training data storage unit 54. Alternatively, the preprocessing unit 230 may be configured separately as a preprocessing unit for CT images and a preprocessing unit for MR images.

[0079] The learning model 244 is composed of a first generator 220G, a coordinate information merging unit 222, a first discriminant 224D, a second generator 250F, a coordinate information merging unit 256, and a second discriminant 266D.

[0080] The first generator 220G and the second generator 250F are each constructed using a 3D CNN. The network structures of the first generator 220G and the second generator 250F may be the same as those of the generator 20G described in the first embodiment.

[0081] The network structures of the first discriminant 224D and the second discriminant 266D may be the same as those of the discriminant 24D described in the first embodiment.

[0082] The first generator 220G is a 3D generator that performs CT→MR domain conversion. It accepts 3D data with the characteristics of the CT domain as input and generates and outputs 3D data with the characteristics of the MR domain. In Figure 10, the notation "3D_CT" input to the first generator 220G represents the 3D data of an actual CT image.

[0083] The coordinate information merging unit 222 merges the coordinate information channels (3ch) with the pseudo-MR image generated by the first generator 220G. The coordinate information merged with the pseudo-MR image is the coordinate information attached to the original input image, the actual CT image, before conversion. In Figure 10, the notation "[x,y,z]ct" represents the coordinate information attached to the actual CT image before conversion.

[0084] The first discriminator 224D is an MR discriminator that determines the authenticity of images related to the MR domain. Specifically, the first discriminator 224D is input either data in which a pseudo-MR image generated by the first generator 220G is combined with its corresponding coordinate information, or data in which a real MR image (training data) is combined with its corresponding coordinate information. The first discriminator 224D then determines whether the image is real or a fake image generated by the first generator 220G. In Figure 10, the notation "3D_MR+[x,y,z]mr" represents 4-channel data in which a real MR image (training data) is combined with its corresponding coordinate information.

[0085] The second generator 250F is a 3D generator that performs MR→CT domain conversion. It accepts 3D data with the characteristics of the MR domain as input and generates and outputs 3D data with the characteristics of the CT domain. In Figure 10, the notation "3D_MR" input to the second generator 250F represents the 3D data of an actual MR image.

[0086] The coordinate information merging unit 256 merges the coordinate information channels (3ch) with the pseudo-CT image generated by the second generator 250F. The coordinate information merged with the pseudo-CT image is the coordinate information attached to the original input image, the actual MR image, before conversion. The notation "[x,y,z]mr" in Figure 10 represents the coordinate information attached to the actual MR image before conversion.

[0087] The second discriminator 266D is a CT discriminator that determines the authenticity of images related to the CT domain. Specifically, the second discriminator 266D is input either data in which a pseudo-CT image and its corresponding coordinate information are combined, or data in which a real CT image and its corresponding coordinate information are combined as training data. The second discriminator 266D then determines whether the image is real or a fake image generated by the second generator 250F. In Figure 10, the notation "3D_CT+[x,y,z]ct" represents 4-channel data in which a real CT image (which is training data) and its corresponding coordinate information are combined.

[0088] Furthermore, the output of the first generator 220G can be input to the second generator 250F. The image after CT→MR conversion by the first generator 220G is further converted from MR to CT by the second generator 250F to generate a reconstructed image (reconstructed pseudo-CT image). Similarly, the output of the second generator 250F can be input to the first generator 220G. The image after MR→CT conversion by the second generator 250F is further converted from CT to MR by the first generator 220G to generate a reconstructed image (reconstructed pseudo-MR image).

[0089] The error calculation unit 246 evaluates the error (Adversarial Loss) between the output from each discriminator (224D, 266D) and the correct answer using a loss function. Furthermore, the error calculation unit 246 evaluates the reconstruction loss (Cycle Consistency Loss) due to image transformation by connecting the first generator 220G and the second generator 250F.

[0090] Reconstruction loss includes the error between the reconstructed image output from the second generator 250F (by inputting the output of the CT→MR conversion from the first generator 220G to the second generator 250F) and the original input image input to the first generator 220G (reconstruction loss due to CT→MR→CT conversion), and the output of the MR→CT conversion from the second generator 250F. 1 generator 2 2 0 GBy inputting the data into the first generator 220G, there is an error between the reconstructed generated image output from the first generator 220G and the original input image input to the second generator 250F (reconstruction loss due to MR→CT→MR conversion).

[0091] The optimizer 248 updates the network parameters in the learning model 244 based on the calculation results of the error calculation unit 246. The optimizer 248 performs a parameter calculation process to calculate the amount of update for the parameters of each network, the first generator 220G, the first discriminant 224D, the second generator 250F, and the second discriminant 266D, based on the calculation results of the error calculation unit 46, and a parameter update process to update the parameters of each network according to the calculation results of the parameter calculation process.

[0092] <Overview of CT input processing (CT → MR)> Figure 11 is a schematic diagram showing the processing flow when CT input occurs in the machine learning system 210 according to the second embodiment. The CT image CTr, which is 3D data belonging to the training dataset of Domain A, is input to the first generator 220G. The first generator 220G receives the input of the CT image CTr, performs CT→MR conversion, and outputs a pseudo-MR image MRsyn that has the characteristics of Domain B.

[0093] For this pseudo-MR image MRsyn, coordinate information including the x, y, and z coordinate data associated with the source CT image CTr is combined as a new channel, and the four-channel data including the pseudo-MR image MRsyn and the coordinate information is input to the first discriminator 224D. The first discriminator 224D is also input four-channel data including the real MR image MRr and its coordinate information. The MR image MRr is three-dimensional data belonging to the training dataset of Domain B. The MR image MRr and the coordinate information including the x, y, and z coordinate data associated with it are combined and input to the first discriminator 224D. The first discriminator 224D performs convolution on the input four-channel data to determine the authenticity of the image. Based on the discrimination result of the first discriminator 224D, the adversarial loss is calculated.

[0094] Furthermore, the pseudo-MR image MRsyn generated by the first generator 220G is input to the second generator 250F, which, upon receiving the pseudo-MR image MRsyn, performs an MR to CT conversion and outputs a reconstructed pseudo-CT image CTsynrec that possesses the characteristics of domain A.

[0095] The machine learning system 210 evaluates the reconstruction loss, which represents the difference between the reconstructed pseudo-CT image CTsynrec output from the second generator 250F and the original CT image CTr. This reconstruction loss is an example of the "first reconstruction loss" in this disclosure.

[0096] The reconstructed pseudo-CT image CTsynrec, generated by a conversion process using the first generator 220G and the second generator 250F in this order, is an example of the "first reconstructed generated image" in this disclosure.

[0097] <Overview of processing during MR input (MR → CT)> Figure 12 is a schematic diagram showing the processing flow when an MR image is input in the machine learning system 210 according to the second embodiment. The MR image MRr, which is 3D data belonging to the training dataset of Domain B, is input to the second generator 250F. The second generator 250F receives the input of a CT image CTr, performs a CT→MR conversion, and outputs a pseudo-CT image CTsyn that has the characteristics of Domain A.

[0098] For this pseudo-CT image CTsyn, coordinate information including the x, y, and z coordinate data associated with the source MR image MRr is combined as a new channel, and the four-channel data including the pseudo-CT image CTsyn and the coordinate information is input to the second discriminator 266D. The second discriminator 266D is also input four-channel data including the actual CT image CTr and its coordinate information. The CT image CTr is three-dimensional data belonging to the training dataset of Domain A. The CT image CTr and the coordinate information including the x, y, and z coordinate data associated with it are combined and input to the second discriminator 266D. The second discriminator 266D performs convolution on the input four-channel data to determine the authenticity of the image. Adversarial loss is calculated based on the discrimination result of the second discriminator 266D.

[0099] Furthermore, the pseudo-CT image CTsyn generated by the second generator 250F is input to the first generator 220G, which, upon receiving the pseudo-CT image CTsyn, performs a CT→MR conversion and outputs a reconstructed pseudo-MR image MRsynrec that possesses the characteristics of domain B.

[0100] In the machine learning system 210, the reconstruction loss, which shows the difference between the reconstructed pseudo-MR image MRsynrec output from the first generator 220G and the original MR image MRr, is evaluated. This reconstruction loss is an example of the "second reconstruction loss" in this disclosure. The reconstructed pseudo-MR image MRsynrec generated by a conversion process using the second generator 250F and the first generator 220G in that order is an example of the "second reconstructed generated image" in this disclosure.

[0101] The 3D CNN used in the second generator 250F of the second embodiment is an example of the "third convolutional neural network" in this disclosure. The pseudo-CT image CTsyn generated by the second generator 250F is an example of the "second generated image" in this disclosure. The 3D CNN used in the second discriminator 266D is an example of the "fourth convolutional neural network" in this disclosure. The image data input to the second discriminator 266D is an example of the "second image data" in this disclosure.

[0102] <Effects of the second embodiment> By performing training using the machine learning system 210 according to the second embodiment, the first generator 220G can acquire the ability to generate CT→MR conversion images and become a 3D image converter that generates high-quality pseudo-MR images. Similarly, the second generator 250F can acquire the ability to generate MR→CT conversion images and become a 3D image converter that generates high-quality pseudo-CT images.

[0103] Example of a machine learning system configuration Figure 13 is a block diagram showing an example configuration of an information processing device 400 applied to a machine learning system 10,210. The information processing device 400 comprises a processor 402, a tangible, non-temporary computer-readable medium 404, a communication interface 406, an input / output interface 408, a bus 410, an input device 414, and a display device 416. The processor 402 is an example of a “first processor” in this disclosure. The computer-readable medium 404 is an example of a “first storage device” in this disclosure.

[0104] The processor 402 includes a CPU (Central Processing Unit). The processor 402 may also include a GPU (Graphics Processing Unit). The processor 402 is connected to a computer-readable medium 404, a communication interface 406, and an input / output interface 408 via a bus 410. An input device 414 and a display device 416 are connected to the bus 410 via the input / output interface 408.

[0105] The computer-readable medium 404 includes memory, which is the main memory, and storage, which is the secondary memory. The computer-readable medium 404 may be, for example, semiconductor memory, a hard disk drive (HDD), or a solid state drive (SSD), or a combination of these.

[0106] The information processing device 400 is connected to a telecommunications line (not shown) via a communication interface 406. The telecommunications line may be a wide-area communication line, a local area communication line, or a combination of these.

[0107] The computer-readable medium 404 stores multiple programs and data for various processing tasks. For example, the computer-readable medium 404 stores a training data generation program 420 and a training processing program 430. The training data generation program 420 may include a coordinate information generation program 422 and a cropping program 424. The training processing program 430 may include a training model 244, an error calculation program 436, and a parameter update program 438. The training model 44 may be used instead of the training model 244. The training data generation program 420 may be incorporated into the training processing program 430.

[0108] By executing the instructions of these programs, the information processing device 400, including the processor 402, functions as a processing unit corresponding to the program. For example, by executing the instructions of the coordinate information generation program 422, the processor 402 functions as a coordinate information generation unit 33 that generates coordinate information of the human body coordinate system. Also, by executing the instructions of the learning processing program 430, the processor 402 functions as a learning processing unit 40,240 that performs learning processing. The same applies to other programs. A portion of the storage area of ​​the computer-readable medium 404 may function as a learning data storage unit 54.

[0109] Furthermore, a computer-readable medium 404 stores a display control program (not shown). The display control program generates display signals necessary for display output to the display device 416 and controls the display of the display device 416.

[0110] The display device 416 is comprised of, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. The input device 414 is comprised of, for example, a keyboard, a mouse, a multi-touch panel, or other pointing device, or an audio input device, or an appropriate combination thereof. The input device 414 accepts various inputs from the operator.

[0111] Examples of medical image processing devices using pre-trained models Figure 14 is a block diagram showing an example configuration of a medical image processing device 500 to which a trained model generated by performing a training process using the machine learning system 10,210 is applied.

[0112] The medical image processing device 500 comprises a processor 502, a tangible, non-temporary computer-readable medium 504, a communication interface 506, an input / output interface 508, a bus 510, an input device 514, and a display device 516.

[0113] The hardware configuration, including the processor 502, computer-readable medium 504, communication interface 506, input / output interface 508, bus 510, input device 514, and display device 516, may be the same as the corresponding elements of the processor 402, computer-readable medium 404, communication interface 406, input / output interface 408, bus 410, input device 414, and display device 416 in the information processing device 400 described in Figure 13. The processor 502 is an example of a "second processor" in this disclosure. The "computer-readable medium 504" is an example of a "second storage device" in this disclosure.

[0114] The computer-readable medium 504 of the medical image processing device 500 stores at least one of the following: a CT-MR conversion program 520 and an MR-CT conversion program 530. The CT-MR conversion program 520 includes a trained generator 522 that has learned CT→MR domain conversion. The trained generator 522 is a trained model corresponding to the generator 20G in Figure 5 or the first generator 220G in Figure 12. The trained generator 522 is an example of the “first trained model” in this disclosure. The CT image input to the first generator 220G is an example of the “first medical image” in this disclosure. The pseudo-MR image output from the first generator 220G is an example of the “second medical image” in this disclosure. The pseudo-MR image output from the trained generator 522 is an example of the “second medical image” in this disclosure.

[0115] The MR-CT conversion program 530 includes a trained generator 532 that has learned the MR→CT domain conversion. The trained generator 532 is a trained model corresponding to the second generator 250F in Figure 12.

[0116] The computer-readable medium 504 may further include at least one of the following programs: an organ recognition AI program 540, a disease detection AI program 542, and a report creation support program 544.

[0117] The organ recognition AI program 540 includes a processing module for organ segmentation. The organ recognition AI program 540 may include a lung segment labeling program, a vascular region extraction program, and a bone labeling program, among others.

[0118] The disease detection AI program 542 includes a detection processing module corresponding to a specific disease. The disease detection AI program 542 may include, for example, at least one of the following programs: a lung nodule detection program, a lung nodule characterization program, a pneumonia CAD program, a breast CAD program, a liver CAD program, a brain CAD program, and a colon CAD program.

[0119] The report creation support program 544 includes a trained document generation model that generates candidate findings statements corresponding to the target medical image.

[0120] Various processing programs, such as the organ recognition AI program 540, the disease detection AI program 542, and the report creation support program 544, may be AI processing modules that include a trained model trained to obtain the output of the target task by applying machine learning such as deep learning.

[0121] AI models for CAD can be constructed, for example, using various CNNs with convolutional layers. Input data for the AI ​​model may include medical images such as 2D images, 3D images, or moving images, and the output from the AI ​​model may be, for example, information indicating the location of disease areas (lesion sites) within the image, or information indicating classification such as disease names, or a combination of these.

[0122] AI models that handle time-series data and document data can be constructed using, for example, various types of recurrent neural networks (RNNs). Time-series data includes, for example, electrocardiogram waveform data. Document data includes, for example, medical reports written by doctors.

[0123] The generated images produced by the CT-MR conversion program 520 or the MR-CT conversion program 530 can be input into at least one of the following programs: the organ recognition AI program 540, the disease detection AI program 542, and the report creation support program 544. This makes it possible to apply AI processing modules built for a specific modality to images from other modalities, thereby expanding the range of applications.

[0124] Variation 3 In the second embodiment, a CycleGAN-based learning framework was adopted, but it is not limited to this. For example, it is possible to modify the input to the discriminator by using StarGAN, which performs multimodality transformations, or MUNIT (Multimodal Unsupervised Image-to-image Translation), and to introduce coordinate information obtained from the human body coordinate system into the learning process. A document describing StarGAN is "Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, Jaegul Choo, "StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation" arxiv:1711.09020". A document describing MUNIT is "Xun Huang, Ming-Yu Liu, Serge Belongie, Jan Kautz, "Multimodal Unsupervised Image-to-Image Translation" arxiv:1804.04732".

[0125] 《Modification 4》 The technology disclosed herein can be applied to various types of image data. CT images may include contrast-enhanced CT images acquired using a contrast agent and non-contrast CT images acquired without a contrast agent. MR images may include T1-weighted images, EOB-enhanced images, non-contrast images, in-phase images, out-of-phase images, T2-weighted images, and fat-suppressed images. EOB is an MRI contrast agent containing sodium gadoxetate (Gd-EOB-DTPA).

[0126] While we have described the image generation task between heterogeneous modalities CT and MR as an example of domain conversion, the technology disclosed can be applied to other examples of domain conversion besides CT-MR, such as conversion tasks to different imaging parameters within MR (e.g., T1-weighted to T2-weighted), or conversion between contrast-enhanced and non-contrast-enhanced images within CT.

[0127] 《About the types of 3D images》 The technology disclosed herein is not limited to CT and MR images, but can also be applied to various medical images acquired by various medical devices, such as ultrasound images that project human body information and PET images acquired using positron emission tomography (PET) equipment.

[0128] Examples of computer hardware configurations Figure 15 is a block diagram showing an example of a computer hardware configuration. Computer 800 may be a personal computer, a workstation, or a server computer. Computer 800 can be used as a device that includes all or part of the machine learning system 10,210 and the medical image processing device 500, or a combination of these functions, as described earlier.

[0129] The computer 800 includes a CPU 802, RAM (Random Access Memory) 804, ROM (Read Only Memory) 806, GPU 808, storage 810, communication unit 812, input device 814, display device 816, and bus 818. The GPU 808 may be provided as needed.

[0130] The CPU 802 reads various programs stored in the ROM 806 or storage 810, etc., and executes various processes. The RAM 804 is used as the CPU 802's workspace. The RAM 804 is also used as a storage unit to temporarily store the read programs and various data.

[0131] The storage 810 includes, for example, a storage device configured using a hard disk drive, an optical disk, a magneto-optical disk, or a semiconductor memory, or an appropriate combination thereof. Various programs and data are stored in the storage 810. When a program stored in the storage 810 is loaded into the RAM 804 and executed by the CPU 802, the computer 800 functions as a means of performing various processes defined by the program.

[0132] The communication unit 812 is an interface that performs communication processing with external devices via wired or wireless connections and exchanges information with external devices. The communication unit 812 can also act as an information acquisition unit that accepts input such as images.

[0133] The input device 814 is an input interface that accepts various operation inputs to the computer 800. The input device 814 may be, for example, a keyboard, mouse, multitouch panel, or other pointing device, or an audio input device, or an appropriate combination thereof.

[0134] The display device 816 is an output interface on which various types of information are displayed. The display device 816 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof.

[0135] Regarding programs that operate computers: It is possible to record a program that implements at least one part or all of the various processing functions, such as the data acquisition function, preprocessing function, and learning processing function in the machine learning systems 10,210 described in the above embodiments, and the image processing function in the medical image processing device 500, on a computer-readable medium which is a tangible non-temporary information storage medium such as an optical disk, magnetic disk, or semiconductor memory, and provide the program through this information storage medium.

[0136] Alternatively, instead of providing programs by storing them on tangible, non-temporary computer-readable media, it is also possible to provide program signals as a download service using telecommunication lines such as the Internet.

[0137] Furthermore, at least one of the various processing functions, such as the data acquisition function, preprocessing function, and learning processing function in the machine learning system 10,210, and the image processing function in the medical image processing device 500, may be implemented by cloud computing. a It can also be offered as a Software as a Service (S).

[0138] Regarding the hardware configuration of each processing unit: The hardware structure of the processing unit that executes various processes such as the generator 20G, coordinate information merging unit 22, discriminator 24D, learning data generation unit 30, coordinate information generation unit 33, cropping processing unit 34, data acquisition unit 42, learning processing units 40, 240, error calculation units 46, 246, optimizers 48, 248, preprocessing unit 230, first generator 220G, second generator 250F, coordinate information merging units 222, 256, first discriminator 224D, and second discriminator 266D is, for example, various processors as shown below.

[0139] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units; GPUs, which are processors specialized for image processing; Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing; and Dedicated Electrical Circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform particular processing.

[0140] A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers such as clients and servers. Secondly, a configuration where a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of System-on-a-Chip (SoC) systems. Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors.

[0141] Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0142] "others" The embodiments of the present invention described above can be modified, added to, or deleted as appropriate without departing from the spirit of the invention. The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention by those with ordinary skill in the art. [Explanation of Symbols]

[0143] 10 Machine Learning Systems 20G generator 22 Coordinate Information Integration Section 24D discriminator 30. Training Data Generation Unit 33 Coordinate information generation section 34. Crop Processing Unit 40 Learning Processing Unit 42 Data Acquisition Unit 44 Learning Models 46 Error calculation section 48 Optimizer 50 Image storage section 54. Learning Data Storage Unit 210 Machine Learning Systems 220G 1st generator 222 Coordinate Information Integration Section 224D 1st discriminator 230 Pre-processing section 240 Learning Processing Unit 244 Learning Models 246 Error calculation section 248 Optimizer 250F 2nd generator 256 Coordinate Information Integration Section 266D 2nd discriminator 400 Information Processing Devices 402 Processors 404 Computer-readable media 406 Communication Interface 408 Input / Output Interfaces 410 Bus 414 Input device 416 Display device 420 Training Data Generation Program 422 Coordinate Information Generation Program 424 Cropping Program 430 Learning Processing Program 436 Error Calculation Program 438 Parameter Update Program 500 Medical imaging processing equipment 502 Processors 504 Computer-readable media 506 Communication Interface 508 Input / Output Interfaces 510 Bus 514 Input device 516 Display device 520 CT-MR Conversion Program 522 Pre-trained generators 530 MR-CT Conversion Program 532 Pre-trained generators 540 Organ Recognition AI Program 542 Disease Detection AI Program 544 Report Writing Support Program 800 Computers 802 CPU 804 RAM 806 ROM 808 GPU 810 storage 812 Communications Department 814 Input device 816 Display device 818 Bus CTr CT images MR MR image MRsyn Pseudo-MR Images CTsynrec Reconstructed pseudo CT image CTsyn pseudo CT image MRsynrec Reconstructed pseudo MR image

Claims

1. A method for generating a trained model that transforms the domain of an input medical image and outputs a generated image of a different domain, A first generator is configured using a first convolutional neural network that accepts medical image input from a first domain and outputs a first generated image from a second domain different from the first domain, A first discriminator configured using a second convolutional neural network accepts data input including first image data, which is either the first generated image generated by the first generator or a medical image of the second domain included in the training dataset, and coordinate information indicating coordinate values ​​corresponding to the positions of each of a plurality of unit elements constituting the first image data, which indicates coordinate values ​​in a human body coordinate system, which is an absolute coordinate system defined based on the anatomical positions of parts of the human body, and determines the authenticity of the input image. Using a learning model that has the structure of a generative adversarial network including, Computers Multiple training data sets are obtained, including medical images from the first domain and medical images from the second domain. This includes performing a learning process to train the first generator and the first discriminator in an adversarial manner based on the aforementioned plurality of training data, Method for generating a pre-trained model.

2. When the first generated image is input to the first discriminator, the coordinate information corresponding to the first generated image is coordinate information that indicates the coordinate values ​​of the human body coordinate system determined in accordance with the positions of each of the multiple unit elements constituting the medical image of the first domain, which is the source image input to the first generator when generating the first generated image. A method for generating a trained model according to claim 1.

3. The first image data is three-dimensional data, The coordinate information includes x-coordinate information, y-coordinate information, and z-coordinate information that specify the position of each voxel, which is the unit element, in three-dimensional space. The x-coordinate information, y-coordinate information, and z-coordinate information are used as channels and combined with the channels of the first image data or the feature map of the first image data and provided to the first discriminator. A method for generating a trained model according to claim 1 or 2.

4. For each of the medical images used as training data, the coordinate information indicating the position of each unit element in the image is associated with it. A method for generating a trained model according to any one of claims 1 to 3.

5. The aforementioned computer, This includes generating coordinate information corresponding to each of the unit elements in the image for each of the medical images used as training data. A method for generating a trained model according to claim 4.

6. The aforementioned second convolutional neural network includes an input layer, an intermediate layer, and an output layer, wherein the intermediate layer includes a plurality of layers that perform convolution processing. The coordinate information is input to one of the intermediate layers of the second convolutional neural network. A method for generating a trained model according to any one of claims 1 to 5.

7. The aforementioned learning model further, A second generator configured using a third convolutional neural network that accepts medical image input from the second domain and outputs a second generated image from the first domain, The system includes a second discriminant configured using a fourth convolutional neural network that accepts input data including a second image data which is either a second generated image generated by the second generator or a medical image of the first domain included in a training dataset, and coordinate information of the human body coordinate system which indicates coordinate values ​​corresponding to the positions of each of the multiple unit elements constituting the second image data, and determines the authenticity of the input image, The learning process includes a process for training the second generator and the second discriminator in an adversarial manner. A method for generating a trained model according to any one of claims 1 to 6.

8. When the second generated image is input to the second discriminator, the coordinate information corresponding to the second generated image is coordinate information that indicates the coordinate values ​​of the human body coordinate system, which are determined in accordance with the positions of each of the multiple unit elements constituting the medical image of the second domain, which is the source image input to the second generator when generating the second generated image. A method for generating a trained model according to claim 7.

9. The aforementioned computer, A process to calculate the first reconstruction loss of a conversion process using the first and second generators in that order, based on the first reconstructed generated image output from the second generator by inputting the first generated image of the second domain output from the first generator to the second generator, The process includes inputting the second generated image of the first domain output from the second generator into the first generator, and then calculating the second reconstruction loss of the conversion process using the second generator and the first generator in that order, based on the second reconstructed generated image output from the first generator. A method for generating a trained model according to claim 7 or 8.

10. The medical image in the first domain is a first modality image taken using a first modality, which is a medical device. The medical images in the second domain are second modality images taken using a second modality, which is a different type of medical device from the first modality. The learning model is trained to receive the first modality image as input and generate a pseudo-second modality generated image that has the features of an image taken using the second modality. A method for generating a trained model according to any one of claims 1 to 9.

11. A machine learning system for training a learning model that transforms the domain of an input medical image to generate a generated image of a different domain, At least one first processor, The system comprises at least one first storage device in which a program executed by the at least one first processor is stored, The aforementioned learning model, A first generator is configured using a first convolutional neural network that accepts medical image input from a first domain and outputs a first generated image from a second domain different from the first domain, A first discriminator configured using a second convolutional neural network accepts data input including first image data, which is either the first generated image generated by the first generator or a medical image of the second domain included in the training dataset, and coordinate information indicating coordinate values ​​corresponding to the positions of each of a plurality of unit elements constituting the first image data, which indicates coordinate values ​​in a human body coordinate system, which is an absolute coordinate system defined based on the anatomical positions of parts of the human body, and determines the authenticity of the input image. It has the structure of an adversarial generative network that includes, The at least one first processor executes the instructions of the program, Multiple training data sets are obtained, including medical images from the first domain and medical images from the second domain. Based on the aforementioned plurality of training data, a learning process is performed to train the first generator and the first discriminator in an adversarial manner. Machine learning system.

12. A program that causes a computer to perform a process of training a learning model that transforms the domain of an input medical image to generate a generated image of a different domain, The aforementioned learning model, A first generator is configured using a first convolutional neural network that accepts medical image input from a first domain and outputs a first generated image from a second domain different from the first domain, A first discriminator configured using a second convolutional neural network accepts data input including first image data, which is either the first generated image generated by the first generator or a medical image of the second domain included in the training dataset, and coordinate information indicating coordinate values ​​corresponding to the positions of each of a plurality of unit elements constituting the first image data, which indicates coordinate values ​​in a human body coordinate system, which is an absolute coordinate system defined based on the anatomical positions of parts of the human body, and determines the authenticity of the input image. It has the structure of an adversarial generative network that includes, To the aforementioned computer, Multiple training data sets are obtained, including medical images from the first domain and medical images from the second domain. Based on the aforementioned plurality of training data, a learning process is executed to train the first generator and the first discriminator in an adversarial manner. program.

13. A non-temporary and computer-readable recording medium on which the program described in claim 12 is recorded.

Images