Pseudo-data generation device, pseudo-data generation method, pseudo-data generation program, learning device, learning method, and learning program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- CANON KK
- Filing Date
- 2022-11-02
- Publication Date
- 2026-08-03
Smart Images
Figure 0007899049000001 
Figure 0007899049000002 
Figure 0007899049000003
Abstract
Description
Technical Field
[0001] The embodiments disclosed in this specification and the drawings relate to a pseudo-data generation device, a pseudo-data generation method, a pseudo-data generation program, a learning device, a learning method, and a learning program.
Background Art
[0002] Machine learning such as deep neural networks has been applied in many fields, and various methods have also been applied in the medical field. In machine learning, since it is premised on learning using a large amount of data, there is a problem that the expected performance cannot be obtained if the number of data is insufficient. Especially in the medical field, it is difficult to collect a large number of various medical data including medical images from the perspective of privacy protection and the like. Also, for example, in the medical field, there is medical data that is not easy to acquire or physically impossible to acquire, such as 3D volume MR (Magnetic Resonance) dynamic images (cine images). If such medical data can be used as correct data or the like, the scope of application of machine learning will be expanded, so there is a need to use such medical data.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Non-Patent Documents
[0004]
Non-Patent Document 1
[0005] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is the generation of pseudo-data of data that is not easily observed. However, the problems that the embodiments disclosed herein and in the drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]
[0006] The pseudo-data generation device according to this embodiment includes an acquisition unit and a generation unit. The acquisition unit acquires one or more partial observation data that may be part of the total observation data. The generation unit generates pseudo-total observation data, which is pseudo-data of the total observation data, by inputting the one or more partial observation data into a function. The function is optimized by converting the pseudo-total observation data for training to calculate pseudo-partial observation data for training, and by learning so that the partial observation data for training and the pseudo-partial observation data are similar. [Brief explanation of the drawing]
[0007] [Figure 1] Figure 1 is a block diagram showing a learning device and a pseudo-data generation device according to the first embodiment. [Figure 2] Figure 2 is a flowchart showing the learning process of the learning device according to the first embodiment. [Figure 3] Figure 3 is a conceptual diagram showing the generation process according to the first embodiment. [Figure 4] Figure 4 is a conceptual diagram showing the identification process according to the first embodiment. [Figure 5] Figure 5 is a flowchart showing the learning process of the learning device according to the second embodiment. [Figure 6] Figure 6 is a conceptual diagram showing the generation process according to the second embodiment. [Figure 7] Figure 7 is a conceptual diagram showing the identification process according to the second embodiment. [Figure 8] Figure 8 is a conceptual diagram showing a modified example of the generation process according to the second embodiment. [Figure 9] Figure 9 shows an example of generating partial observation data. [Modes for carrying out the invention]
[0008] Hereinafter, the pseudo-data generation device, pseudo-data generation method, pseudo-data generation program, learning device, learning method, and learning program according to this embodiment will be described with reference to the drawings. In the following embodiments, parts with the same reference numerals perform similar operations, and redundant explanations will be omitted as appropriate. Hereinafter, one embodiment will be described with reference to the drawings.
[0009] (First embodiment) A learning device and a pseudo-data generation device according to the first embodiment will be described with reference to the block diagram in Figure 1. The learning device 1 according to the first embodiment includes a processing circuit 2, an input interface 4, a communication interface 6, and a memory 8. The processing circuit 2 includes an acquisition function 21, a generation function 22, a conversion function 23, an identification function 24, and a learning function 25. The processing circuit 2 has a processor as a hardware resource (not shown).
[0010] The acquisition function 21 acquires multiple partial observation data that can become part of the total observation data. Total observation data refers to data that is not easy to acquire. For example, in an MRI (Magnetic Resonance Imaging) device, acquiring MR moving images of a 3D volume is not easy. On the other hand, partial observation data that corresponds to a part of the composition of data that is not easy to acquire can often be acquired relatively easily. For example, 2D moving images or 3D volume images (still images) relating to slice cross-sections can be acquired with normal imaging.
[0011] The generation function 22 generates pseudo-whole observation data, which is pseudo-data of the whole observation data, from multiple partially observed data. Here, it is assumed that a generator (also called a generation model or function) of a Conditional Generative Adversarial Network (cGAN) is used. Note that, in addition to cGAN, a decoder trained using a Conditional Variational Auto Encoder (cVAE) or a trained model trained using a conditional diffusion model may also be used. The conversion function 23 converts the pseudo-whole observation data into pseudo-first partial observation data, which is pseudo-data of the first partial observation data among multiple partial observation data.
[0012] The discrimination function 24 evaluates a value that identifies whether the first partial observation data is true data or pseudo-data, based on the pseudo-first partial observation data or the first partial observation data and the partial observation data other than the first partial observation data among the multiple partial observation data, using a loss function. Here, for example, it is assumed that a cGAN classifier (also called a discriminative model) is used. If both pseudo-first partial observation data and first partial observation data are available during the learning process, for example, the squared error between the pseudo-first partial observation data and the first partial observation data may be evaluated using the loss function, or for example, the weighted sum of the output value of the cGAN classifier and the output value of the squared error may be evaluated using the loss function. The learning function 25 optimizes the parameters of the generator used by the generation function 22 and the discriminator used by the discrimination function 24 so that the pseudo first partial observation data is determined as the true data by the discriminator. For example, when using the squared error in the learning function 25, the parameters of the generator used by the generation function 22 are optimized so that the pseudo first partial observation data is similar to the first partial observation data.
[0013] The input interface 4 has a circuit for receiving various instructions and information inputs from the user. The input interface 4 has, for example, a circuit related to a pointing device such as a mouse or an input device such as a keyboard. Note that the circuit included in the input interface 4 is not limited to a circuit related to physical operation components such as a mouse and a keyboard. For example, the input interface 4 may have an electric signal processing circuit that receives an electric signal corresponding to an input operation from an external input device provided separately from the learning device 1 and outputs the received electric signal to various circuits in the learning device 1 or the pseudo data generation device.
[0014] The communication interface 6 exchanges data with an external device by wire or wirelessly. For example, it performs data communication with a workstation, a RIS (Radiology Information systems) server, and a PACS (Picture Archiving and Communication Systems) server.
[0015] The memory 8 stores data sets such as partial observation data and pseudo full observation data, and a learned model. The memory 8 is a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disk, or the like. Further, the memory 8 may be a drive device or the like that reads and writes various information to and from a portable storage medium such as a CD-ROM drive, a DVD drive, or a flash memory.
[0016] In addition, when operating as a pseudo-data generation device after learning by the learning device 1, it may have a configuration including a processing circuit 2 including an acquisition function 21 and a generation function 22, an input interface 4, a communication interface 6, and a memory 8. The acquisition function 21 acquires one or more partial observation data. The generation function 22 generates pseudo full-observation data by inputting one or more partial observation data into a function.
[0017] Also, various functions in the processing circuit 2 may be stored in the memory 8 in the form of a program executable by a computer. In this case, it can be said that the processing circuit 2 is a processor that realizes the functions corresponding to each program by reading out and executing the programs corresponding to these various functions from the memory 8. In other words, the processing circuit 2 in the state of having read each program will have a plurality of functions etc. shown in the processing circuit 2 of FIG. 1. In FIG. 1, although it has been described that these various functions are realized by a single processing circuit 2, it is also possible to configure the processing circuit 2 by combining a plurality of independent processors, and each processor realizes the function by executing a program. In other words, it may be the case where each of the above-mentioned functions is configured as a program and one processing circuit executes each program, or it may be the case where a specific function is implemented in a dedicated independent program execution circuit.
[0018] Next, the learning process of the learning device 1 according to the first embodiment will be described with reference to the flowchart of FIG. 2.
[0019] In step S201, the processing circuit 2 acquires a plurality of partial observation data by the acquisition function 21. In step S202, the processing circuit 2 generates pseudo full-observation data using a generator based on a plurality of partial observation data and condition data by the generation function 22.
[0020] In step S203, the processing circuit 2 performs a conversion process on the pseudo-full observation data using the conversion function 23, generating pseudo-partial observation data for each of the multiple partial observation data. Ideally, the conversion process by the conversion function 23 is such that, assuming that the real full observation data is obtained, the converted pseudo-partial observation data generated by converting the real full observation data is identical to the corresponding partial observation data.
[0021] In step S204, the processing circuit 2, using the identification function 24, uses a classifier to determine whether the pseudo-partial observation data is real (i.e., partial observation data) or fake (i.e., data that is not partial observation data) based on the pseudo-partial observation data, partial observation data, and other partial observation data. The identification process by the classifier is performed for each pseudo-partial observation data generated in step S203, for the corresponding partial observation data. The identification process also outputs, for example, the probability (likelihood) value that the pseudo-partial observation data is real from the classifier.
[0022] In step S205, the processing circuit 2 determines whether learning is complete using the learning function 25. For example, if the value of the cGAN loss function, such as a loss function using binary cross-entropy, is below a threshold, it can be determined that learning is complete. Alternatively, if the number of learning epochs exceeds a predetermined number, it can be determined that learning is complete. Any determination criterion used in GANs and machine learning for determining learning completion can be used. If learning is complete, the process ends and the parameters of the generator and discriminator are optimized. If learning is not complete, the process proceeds to step S206.
[0023] In step S206, the parameters (such as weight coefficients) of the generator and classifier are changed, and the process is repeated back to step S203. The generator's parameters are adjusted so that it generates pseudo-partial observation data that the classifier can distinguish (recognize) from the real data (i.e., the partial observation data obtained in step S201). On the other hand, the parameters of the classifier are adjusted so that it can identify the pseudo-partial observation data generated by the generator as fake. As described above, the generator and classifier learn from each other through the learning process shown in Figure 2, and the learning is completed. Note that the identification function 24, as shown in the example in Figure 2, performs a truth determination on whether the pseudo-partial observation data is real or fake, but is not limited to this. For example, the mean squared error or mean absolute error between partial observation data may be output as the loss value, or the weighted sum of the mean squared error (or mean absolute error) and the output value of the classifier may be evaluated using a loss function. For example, the parameters of the generator and classifier should be optimized by the learning function 25 so that the loss value based on the loss function is minimized.
[0024] Next, the generation process by the generation function 22 according to the first embodiment will be explained with reference to the conceptual diagram in Figure 3. Figure 3 shows an example of input and output data for the cGAN generator G used in the first embodiment.
[0025] The generator G receives the first partial observation data S1 corresponding to the first partial observation data, the second partial observation data S2 corresponding to the second partial observation data, a latent variable z, and conditional data C as input. The latent variable z is a variable in the latent space given to the GAN generator G, and here a pseudorandom number within a predetermined range is used. The conditional data C indicates whether the first partial observation data S1 and the second partial observation data S2 are valid or not, and here it is assumed that a one-hot vector is used. A one-hot vector is a vector to which, for example, if the partial observation data input to the generator G exists, it is assigned "1" to indicate that it is valid, and if it does not exist, it is assigned "0" to indicate that it is invalid. The generator G outputs the pseudo-whole observation data Y.
[0026] Next, the identification process by the identification function 24 according to the first embodiment will be explained with reference to the conceptual diagram in Figure 4. Figure 4 shows an example of input and output data for the cGAN classifier D used in the first embodiment. Multiple classifiers D are prepared according to the number of partial data, such as classifier D1 and classifier D2.
[0027] First, the conversion function 23 performs a conversion process P1 on the pseudo-total observation data Y generated by the generator G, generating pseudo-first partial observation data S1'. Similarly, the conversion function 23 performs a conversion process P2 on the pseudo-total observation data Y, generating pseudo-second partial observation data S2'.
[0028] The classifier D1 receives the generated pseudo-first partial observation data S1', the first partial observation data S1, the second partial observation data S2, and conditional data C as input. Here, the conditional data C input to classifier D1 indicates that the second partial observation data S2 is valid. Note that instead of inputting conditional data C, if no data exists, known data (for example, all zeros) can be input as the second partial observation data S2, and processing can be performed regardless of whether the second partial observation data S2 is valid or not, so conditional data C does not need to be input to classifier D1.
[0029] Classifier D1 determines whether the pseudo-first partial observation data S1' is identical to the first partial observation data S1, i.e., genuine, or whether the pseudo-first partial observation data S1' is fake. Here, the second partial observation data S2 is input to refer to the relationship between the partial observation data. As an example to aid understanding, let's assume that the first partial observation data S1 is the upper half of the overall image, and the second partial observation data S2 is the lower half of the overall image. In this case, with the input of the second partial observation data S2, classifier D1 performs classification considering whether the pseudo-first partial observation data S1' and the second partial observation data S2 are well related.
[0030] Similarly, classifier D2 receives the generated pseudo-second partial observation data S2', the second partial observation data S2, the first partial observation data S1, and conditional data C as inputs. The conditional data C input to classifier D2 indicates that the first partial observation data S1 is valid. However, instead of inputting conditional data C, if known data is input as the first partial observation data S1 when no data exists, processing is possible regardless of whether the first partial observation data S1 is valid or not, so conditional data C does not need to be input to classifier D2. Classifier D2 determines whether the pseudo-second partial observation data S2' is identical to the second partial observation data S2, i.e., genuine, or whether the pseudo-second partial observation data S2' is fake.
[0031] In the training of the generator G shown in Figure 3 and the discriminators D1 and D2 shown in Figure 4, the parameters of the generator G and the discriminators D1 and D2 should be mutually optimized so as to satisfy the following conditions.
[0032] The generator G adjusts the parameters for generating pseudo-full observation data Y such that the classifier D1 recognizes the pseudo-first partial observation data S1' as real (i.e., first partial observation data S1), and the classifier D2 recognizes the pseudo-second partial observation data S2' as real (i.e., second partial observation data S2).
[0033] On the other hand, classifier D1 is parameterized to distinguish the pseudo-first partial observation data S1' from fake data (i.e., data that is not the first partial observation data S1). Similarly, classifier D2 is parameterized to distinguish the pseudo-second partial observation data S2' from fake data (i.e., data that is not the second partial observation data S2).
[0034] Once the training of generator G and discriminators D1 and D2 is complete, the pseudo-data generation device can use generator G, as shown in Figure 3, to generate pseudo-full observation data that is similar to all observation data that is not easy to observe. Specifically, the acquisition function 21 of the pseudo-data generation device acquires the first partial observation data S1 and the second partial observation data S2. The generation function 22 of the pseudo-data generation device takes the first partial observation data S1, the second partial observation data S2, the condition data indicating that the first partial observation data S1 and the second partial observation data S2 are valid, and the latent variable z as input to the generator G, which has completed training, and obtains pseudo-full observation data Y as the output of the generator G.
[0035] Furthermore, the above example is not limited to the two types of partial observation data, S1 and S2, but can also be applied when generating pseudo-whole observation data from three or more (three or more types) of partial observation data. For example, if there are three types of partial observation data, in addition to the classifiers D1 and D2 described above, a classifier D3 can be further set to distinguish between the third partial observation data S3 and the pseudo-third partial observation data S3' converted from the pseudo-whole observation data Y.
[0036] Furthermore, if there are three or more types of partial observation data, the partial observation data input to each classifier D may be all of the other partial observation data, or it may be a combination of multiple data. For example, classifier D3 may be input with pseudo-third partial observation data S3' and third partial observation data S3, as well as all of the other partial observation data, namely first partial observation data S1 and second partial observation data S2. Alternatively, depending on the pseudo-whole observation data to be generated, either first partial observation data S1 or second partial observation data S2 may be used. Specifically, if, in the pseudo-whole observation data, third partial observation data S3 is adjacent to second partial observation data S2 but not to first partial observation data S1, then classifier D3 may be input with pseudo-third partial observation data S3', third partial observation data S3, and second partial observation data S2. Furthermore, we may prepare as many classifiers D as there are possible combinations of partial observation data.
[0037] According to the first embodiment described above, a classifier is prepared for each partially observed data set, for example using a cGAN, to identify the partially observed data, which is relatively easy to observe. The pseudo-whole observed data generated from the partially observed data is converted into pseudo-partial observed data, which is pseudo-data for each partially observed data set. Through a learning process, the classifier distinguishes between partially observed data and pseudo-partial observed data for each partially observed data set. The generator and multiple classifiers learn from each other. During inference, the partially observed data is input to the generator after learning is complete, and pseudo-whole observed data can be generated for all observed data that is unobservable or not easy to observe. Furthermore, in applications such as machine learning using images, product testing, and education, when it is not easy to prepare all observational data, or when it is not necessary to use all observational data, pseudo-full observational data can be used as fake data for display purposes.
[0038] (Second embodiment) In the second embodiment, we assume a case where some of the partial observation data necessary to constitute pseudo-total observation data is missing. The configuration of the functional blocks of the learning device 1 and the pseudo-data generation device according to the second embodiment is the same as that of the learning device 1 and the pseudo-data generation device according to the first embodiment.
[0039] The learning process of the learning device according to the second embodiment will be explained with reference to the flowchart in Figure 5. In step S501, the processing circuit 2 acquires partial observation data using the acquisition function 21. Here, we assume a case where some of the partial observation data, which can be used to make up the entire observation data, cannot be acquired. In step S502, the generation function 22 causes the processing circuit 2 to generate pseudo-full observation data using a generator based on the partial observation data and condition data.
[0040] In step S503, the processing circuit 2 performs a conversion process on the pseudo-total observation data using the conversion function 23, generating pseudo-missing partial data, which is pseudo-data of the partial observation data that could not be obtained. In step S504, the processing circuit 2 uses the identification function 24 to identify, based on the pseudo-missing partial data and the partially observed data, whether the pseudo-missing partial data is fake (i.e., whether it is data that is not partially observed data).
[0041] In step S505, the processing circuit 2 uses the identification function 24 to identify that the pseudo-missing partial data is fake, based on the pseudo-missing partial data and using a classifier, without using the partially observed data. In step S506, the processing circuit 2 performs a conversion process on the pseudo-full observation data using the conversion function 23 to generate pseudo-partial observation data.
[0042] In step S507, the processing circuit 2 uses the identification function 24 to determine, based on the pseudo-partial observation data and the actual partial observation data, whether the pseudo-partial observation data is real (partial observation data) or fake (data that is not partial observation data). In step S508, the processing circuit 2 determines whether learning is complete using the learning function 25. The determination of whether learning is complete is the same as in step S207 shown in Figure 2. If learning is complete, the process ends and the parameters of the generator and discriminator are optimized. If learning is not complete, the process proceeds to step S509. In step S509, the parameters of the generator and discriminator are changed, and the process is repeated by returning to steps S503 and S506.
[0043] Next, the generation process by the generation function 22 according to the second embodiment will be explained with reference to the conceptual diagram in Figure 6. In the example shown in Figure 6 and Figure 7 (described later), the goal was to obtain the first partial observation data S1 and the second partial observation data S2. However, the first partial observation data S1 could not be obtained, and we assume that the first partial observation data S1 is missing partial data. In Figure 6, the inability to obtain the first partial observation data S1 is shown as a dashed block and crossed out with an "x" (crossout).
[0044] The second partial observation data S2, conditional data C, and latent variable z are input to the generator G. The conditional data C only needs to contain information indicating that the first partial observation data S1 is invalid and the second partial observation data S2 is valid. For example, if the one-hot vector is [S1,S2], it can be represented as [0,1]. The processing of the generator G is largely unchanged from the first embodiment. Without using the missing first partial observation data, pseudo-full observation data Y is generated from the acquired second partial observation data S2, condition data C, and latent variable z.
[0045] Next, the identification process by the identification function 24 according to the second embodiment will be described with reference to Figure 7. In Figure 7, the process is the same as in the first embodiment up to the point where the pseudo-full observation data generated by the generator G shown in Figure 6 is converted to generate pseudo-first partial observation data S1' and pseudo-second partial observation data S2'. In the identification process according to the second embodiment, multiple classifiers D are designed depending on the presence or absence of conditional data input to the classifier D.
[0046] For example, in the case of a classifier that identifies pseudo-first partial observation data S1', the second partial observation data S2, which is input along with it, can be set as condition data C to "S2=1" and "S2=0". Therefore, a classifier D1 is set up that takes the second partial observation data S2 and its condition data C [S2=1 or 0] as input, and a classifier D1' is set up that does not take the condition data C or the second partial observation data S2 as input.
[0047] Specifically, the classifier D1 receives the second partial observation data S2 as input, and the condition data C[S2=1], the acquired second partial observation data S2, and the pseudo-first partial observation data S1' as input. Given the second partial observation data S2, the classifier D1 determines whether the pseudo-first partial observation data S1' is genuine or fake.
[0048] On the other hand, classifier D1' is the classifier for the case where the second partial observation data S2 is not present as input, and furthermore, the first partial observation data S1 is missing, so classifier D1' is input with only the pseudo-first partial observation data S1'. Classifier D1' identifies that the pseudo-first partial observation data S1' is fake. Note that the learning required for classifier D1' to identify as genuine is done using data in which the first partial observation data S1 exists. Classifier D1' is similar to the classifier in a general GAN without any conditions in the case where the correct first partial observation data S1 is not input.
[0049] Similarly, in the case of a classifier that identifies the pseudo-second partial observation data S2', it is possible to set the condition data C for the first partial observation data S1, which is also to be input, as [S1=1] and [S1=0]. Therefore, a classifier D2 is set for when the first partial observation data S1 and its condition data C [S1=1 or 0] are given as input, and a classifier D2' is set for when neither the condition data C nor the first partial observation data S1 are given as input.
[0050] Here, since the first partial observation data S1 is missing, the classifier D2 may be trained to identify the conditional data C[S1=0] and the pseudo-second partial observation data S2' as fake, or it may be trained to identify the conditional data C[S1=0] and the second partial observation data S2 as real. Similarly, the classifier D2' may be trained to identify the pseudo-second partial observation data S2' as fake, or it may be trained to identify the second partial observation data S2 as real. Consequently, the classifier D2' is similar to the classifier that identifies the pseudo-second partial observation data S2' in a general GAN without conditions.
[0051] Although the first partial observation data S1 is missing, classifier D2 will be processed using the condition data C[S1=0], and classifier D2' will be processed using a classifier without any preconditions. Therefore, classifiers D2 and D2' will each be trained using the pseudo-second partial observation data S2'.
[0052] Next, Figure 8 shows a modified example of the identification process according to the second embodiment. As shown in Figure 8, when identifying pseudo-partial observation data from acquired partial observation data, in this case, when identifying pseudo-second partial observation data S2', pseudo-first partial observation data S1' related to the missing first partial observation data may be input to the classifier D2. That is, instead of the first partial observation data S1, pseudo-first partial observation data, which is pseudo-data, may be input, and the classifier D2 may perform the identification process for pseudo-second partial observation data S2'.
[0053] According to the second embodiment described above, even when partial observation data is missing, pseudo-partial observation data is generated from pseudo-full observation data for the missing partial observation data, and the classifier is trained to distinguish between the acquired partial observation data and the pseudo-partial observation data for the missing partial observation data. This makes it possible to generate pseudo-full observation data in the same way as in the first embodiment.
[0054] (Example of partially observed data) Next, specific examples of partial and full observation data according to the first and second embodiments will be described. For example, if the target data is an image, a patch image of the entire image may be used as partial observation data. If the target data is spectral data, partial spectral data of the entire spectrum may be used as partial observation data, or the spectral data as is may be used if it corresponds to a part of the composite spectrum.
[0055] Specifically, when the target data is an image, the learning device and pseudo-data generation device according to the above embodiment can generate a super-resolution image as pseudo-whole observation data from a first-resolution image (low-resolution image) having a first resolution as first-partial observation data and a second-resolution image (high-resolution image) having a second resolution higher than the first resolution as second-partial observation data.
[0056] Next, an example of generating partial observation data when acquiring images and spectra simultaneously will be explained with reference to Figure 9. Figure 9 shows an example of a brain MR image 901 acquired by an MRI machine and an MRS (MR Spectroscopy) spectrum 902. When observing the MR image 901 and spectrum 902 simultaneously, the resolution of the MR image and the resolution of the ROI (Region of Interest) used to collect the MRS spectrum are often different.
[0057] Therefore, as the first method for acquiring partial observation data, the MR image is first segmented. Next, the segmented region is compared with the pixel region of the MRS ROI, and if the pixel region of the MRS ROI is included in the segmented region, then each pixel included in the segmented region is associated with its spectrum.
[0058] Specifically, in Figure 9, segmentation processing is performed on the MR image 901, the tumor area is segmented, and a segmentation region 903 is obtained. Here, since the ROI of the MRS spectrum 902 is included in the segmentation region 903, each pixel in the segmentation region 903 is associated with the spectrum 902. This allows for the generation of partial observation data of the spectrum corresponding to each pixel. If the spectrum corresponding to segmentation region 903 cannot be collected, that spectrum will be treated as missing for that segmentation region.
[0059] Another method for acquiring partial observation data is to obtain multiple MRS spectra at a given ROI. The pixel position of the ROI on the MR image is determined in the MRS, and the pixel regions of the MR image that overlap with the ROI in the MRS are associated with the values of the corresponding spectra. On the other hand, for pixel regions that do not overlap with the ROI in the MRS, the values of the acquired spectra with the closest Euclidean distance can be associated using nearest neighbor interpolation. Note that other interpolation methods such as bilinear interpolation and bicubic interpolation may also be used, not just nearest neighbor interpolation. Furthermore, for spectrum 902, data that has been pre-classified into multiple molecular types, such as choline (Cho) and N-acetylaspartic acid (NAA), may be used.
[0060] The partial observation data described above may be acquired by the processing circuit 2 using the acquisition function 21 after performing the processing described in Figure 9, or the processing circuit 2 may perform the processing described in Figure 9 using a preprocessing function (not shown). In this way, MR images, MR videos, and 1D MRS are acquired as partial observation data. The generation function 22 can generate a 2D CSI (Chemical Shift Imaging) image as pseudo-full observation data if an MR image is acquired, and a CSI video as pseudo-full observation data if an MR video is acquired.
[0061] Furthermore, another example of application to MR images is that pre-contrast 3D images, post-contrast 3D images, and dynamic images related to the contrast curve may be used as partial observation data, and the 3D contrast-reconstructed image may be used as pseudo-full observation data. By training a generator and classifier using this partial observation data, it is possible to obtain 3D contrast-enhanced dynamic images that are not easily acquired.
[0062] Alternatively, weighted images such as T1-weighted images, T2-weighted images, and diffusion-weighted images may be used as partial observation data, and T1 maps (T1 images), T2 maps (T2 images), and diffusion images based on true parameter values may be used as pseudo-whole observation data. By training a generator and a classifier using this partial observation data, it is possible to obtain pseudo-whole observation data that is similar to the true parameter values, which are currently impossible to obtain.
[0063] Furthermore, partial observation data is not limited to MR images; it can also be medical data obtainable from other medical imaging devices. For example, in the case of an X-ray CT (Computed Tomography) device, images of the energy distribution from a photon-counting CT device that discriminates the energy of photons or a dual-energy CT device that uses two types of tube voltages may be used as partial observation data, while material discrimination images of reference substances such as iodine and water may be used as pseudo-full observation data.
[0064] In addition, the system may be connected to any of the following: an X-ray imaging device, a PET (Positron Emission Tomography) device, a SPECT (Single Photon Emission Computed Tomography) device, or an ultrasound diagnostic device. The medical data acquired by the connected medical imaging device may be used as partial observation data, and pseudo-full observation data that can be composed of partial observation data may be generated.
[0065] Of course, partial observation data can be obtained from images that are generally available, such as fixed-point surveillance images from street cameras or microscopic images from microscopes, not just medical data. For example, images from specific viewpoints of the subject being photographed can be used as partial observation data, and a surface-rendered 3D image can be used as pseudo-full observation data. Alternatively, the lens focus can be adjusted to obtain multiple positional images and depth information as partial observation data, and the depth image can be used as pseudo-full observation data. As a result, the pseudo-data generation device, for example, the acquisition function 21, acquires multiple images captured while shifting the focus as multiple partial observation data. The generation function 22 can generate 3D volume data containing depth information as pseudo-observation data from multiple partial observation data.
[0066] (Regularization of all observed data) If there is a shortage of types of partial observation data that make up the total observation data, or if the number of partial observation data is insufficient, the generator and classifier models may overfit. Therefore, to prevent overfitting, regularization may be performed on the entire observation data.
[0067] For example, a penalty term may be added to the loss function of the generator and classifier training to take into account whether the data is smooth or not. Specifically, if the pseudo-whole observation data is smooth, a penalty term should be added that results in a lower Total Variation for the pseudo-whole observation data, in other words, a smaller L2 norm. Furthermore, if all assumed observational data has symmetry in the left-right and / or up-down directions, one can determine whether the pseudo-all observational data is symmetrical in the left-right and / or up-down directions, and add a penalty term that increases in value if it is not symmetrical.
[0068] Additionally, a penalty term may be added to the loss function used for training, such that its value decreases as the error obtained by low-rank approximation of the pseudo-entire set of observed data decreases. Furthermore, if the data is information generated from molecules or atoms, a penalty term may be added to the loss function that decreases as the data is sparser with respect to a particular axis (e.g., the frequency axis), i.e., as the L1 norm becomes smaller.
[0069] Furthermore, data augmentation may be performed to increase the number of data points by deforming or rotating the acquired partial observation data. Regardless of whether it involves deformation or rotation, general methods used by those skilled in the art should be applied to data augmentation. Alternatively, for example, if super-resolution images are assumed as the total observation data, when generating pseudo-partial observation data from the super-resolution images, the pseudo-partial observation data may be generated by shifting it by an amount equivalent to or greater than one pixel in the spatial resolution of the super-resolution image. Specifically, in the conversion process, a window is used to specify the region for generating pseudo-partial observation data from the super-resolution image, and the region on the super-resolution image is cut out while shifting it by one or more pixels at a time. This increases the amount of pseudo-partial observation data.
[0070] The learning device 1 and the pseudo-data generation device may be mounted on at least one of the following: a server, a workstation, or a medical image diagnostic device.
[0071] According to at least one embodiment described above, it is possible to generate pseudo-data of data that is not easily observed.
[0072] In addition, each function according to the embodiment can also be realized by installing a program that performs the processing on a computer such as a workstation and loading it into memory. In this case, the program that can cause the computer to execute the method can also be stored and distributed on a storage medium such as a magnetic disk (hard disk, etc.), optical disk (CD-ROM, DVD, etc.), or semiconductor memory.
[0073] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]
[0074] 1. Learning device 2 Processing Circuit 4 Input Interfaces 6. Communication Interface 8 memory 21 Acquisition function 22 Generation function 23 Conversion function 24. Identification function 25 Learning Functions 901 MR image 902 spectrum 903 Segmentation Region G generator D1,D1',D2,D2',D3 Discriminator
Claims
1. An acquisition unit that acquires one or more partial observation data that may be part of the total observation data, A generation unit generates pseudo-full observation data, which is pseudo-data of the full observation data, by inputting the aforementioned one or more partial observation data into a function, It is equipped with, The aforementioned function is a pseudo-data generator that calculates pseudo-partial observation data for training by converting pseudo-whole observation data for training, and is optimized by learning so that the training partial observation data and the pseudo-partial observation data are similar.
2. The pseudo-data generation device according to claim 1, wherein the function is a generator trained using a conditional adversarial generative network, a decoder trained using a conditional variational autoencoder, or a trained model trained using a conditional spreading model.
3. The acquisition unit acquires multiple images captured while shifting the focus as multiple partial observation data, The pseudo-data generation device according to claim 1, wherein the generation unit generates three-dimensional volume data having depth information from the plurality of partial observation data as the pseudo-total observation data.
4. Obtain one or more partial observational data points that can become part of the total observational data, By inputting the aforementioned one or more partial observation data into the function, pseudo-full observation data, which is pseudo-data of the full observation data, is generated. The function is a pseudo-data generation method that calculates pseudo-partial observation data for training by converting pseudo-whole observation data for training, and is optimized by training so that the training partial observation data and the pseudo-partial observation data are similar.
5. On the computer, An acquisition function that acquires one or more partial observational data that may be part of the total observational data, A generation function is implemented that generates pseudo-full observation data, which is pseudo-data of the full observation data, by inputting one or more of the aforementioned partial observation data into a function. The function is a pseudo-data generation program that calculates pseudo-partial observation data for training by converting pseudo-whole observation data for training, and is optimized by learning so that the training partial observation data and the pseudo-partial observation data are similar.
6. An acquisition unit that acquires multiple partial observation data that can become part of the total observation data, A generation unit that generates pseudo-full observation data, which is pseudo-data of the full observation data, from the plurality of partial observation data, A conversion unit that converts the aforementioned pseudo-total observation data into pseudo-first partial observation data which is pseudo-data of the first partial observation data among the plurality of partial observation data, An identification unit that identifies the pseudo-first partial observation data based on the pseudo-first partial observation data, the first partial observation data, and other partial observation data from the plurality of partial observation data, A learning unit optimizes the parameters of the model used by the generation unit and the identification unit so that the pseudo-first partial observation data is similar to the first partial observation data. A learning device equipped with the following.
7. The plurality of partial observation data include the first partial observation data and a second partial observation data different from the first partial observation data. The conversion unit converts the pseudo-full observation data into pseudo-second partial observation data. The identification unit identifies the pseudo-first partial observation data based on the pseudo-first partial observation data, the first partial observation data, and the second partial observation data, and identifies the pseudo-second partial observation data based on the pseudo-second partial observation data, the second partial observation data, and the first partial observation data. The learning unit optimizes the parameters such that the pseudo-first partial observation data is similar to the first partial observation data, and the pseudo-second partial observation data is similar to the second partial observation data. The learning device according to claim 6.
8. If there are missing partial observation data among the partial observation data that can constitute the total observation data, the conversion unit converts the pseudo-total observation data into pseudo-missing partial data, which are pseudo-data corresponding to the missing partial observation data. The learning device according to claim 6, wherein the learning unit optimizes the parameters using the partially observed data acquired by the acquisition unit and the pseudo-missing partial data.
9. The learning device according to claim 6, wherein the learning unit optimizes the parameters of the generation unit and the identification unit so that regularization is performed on the pseudo-total observation data.
10. The learning device according to claim 9, wherein the regularization is a process that minimizes the L1 norm or L2 norm with respect to the pseudo-total observation data.
11. The learning device according to claim 9, wherein the regularization is achieved by adding a penalty term to the loss function during learning by the learning unit, the value of which decreases as the pseudo-total observed data is more symmetrical in the left-right and / or up-down directions.
12. The aforementioned multiple partial observation data include pre-contrast 3D MR (Magnetic Resonance) images, post-contrast 3D MR images, and video images of the imaging target area. The learning device according to claim 6, wherein all observed data are three-dimensional imaging motion images.
13. The aforementioned multiple partial observation data include MR (Magnetic Resonance) images or MR motion images, and 1D MRS (MR Spectroscopy). The learning device according to claim 6, wherein the total observation data is a two-dimensional CSI (Chemical Shift Imaging) image when an MR image is acquired, and a three-dimensional CSI image when an MR video is acquired.
14. The aforementioned multiple partial observation data include the respective CT (Computed Tomography) images for each energy distribution. The learning device according to claim 6, wherein all observed data are material discrimination images relating to multiple substances.
15. The aforementioned plurality of partial observation data include a first resolution image and a second resolution image having a higher resolution than the first resolution image. The learning device according to claim 6, wherein all observed data are super-resolution images.
16. By acquiring multiple partial observational data that could potentially become part of the total observational data, From the aforementioned multiple partial observation data, pseudo-full observation data, which is pseudo-data of the full observation data, is generated. The aforementioned pseudo-total observation data is converted into pseudo-first partial observation data, which is pseudo-data of the first partial observation data among the plurality of partial observation data. Based on the pseudo-first partial observation data, the first partial observation data, and the other partial observation data among the plurality of partial observation data, the pseudo-first partial observation data is identified. The parameters of the model used in the generation process and the identification process are optimized so that the pseudo-first partial observation data is similar to the first partial observation data. Learning methods.
17. On the computer, A function to acquire multiple partial observation data that can become part of the total observation data, A generation function that generates pseudo-full observation data, which is pseudo-data of the full observation data, from the aforementioned multiple partial observation data, A conversion function that converts the aforementioned pseudo-total observation data into pseudo-first partial observation data, which is pseudo-data of the first partial observation data among the plurality of partial observation data, An identification function that identifies the pseudo-first partial observation data based on the pseudo-first partial observation data, the first partial observation data, and other partial observation data from the plurality of partial observation data, A learning function that optimizes the parameters of the model used in the generation function and the identification function so that the pseudo-first partial observation data is similar to the first partial observation data, A learning program that makes this a reality.