Data processing apparatus and data processing method

Abstract

To generate, at low cost, training data for machine learning, for example, enormous number of training data for generating a machine learning model for segmentation.SOLUTION: A data processing apparatus of an embodiment comprises: a first generation unit that generates first artificial data; a second generation unit that generates second artificial data; and a third generation unit that mixes the first artificial data and the second artificial data to generate mixed data, and generates mixed area information related to an area where the first artificial data and the second artificial data are mixed.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to a data processing device and a data processing method. [Background technology]

[0002] In recent years, the application of AI (Artificial Intelligence) technology, particularly technology using machine learning, has been expanding in various types of image processing including medical image processing and various types of data processing including medical data processing.

[0003] A technique called segmentation is known as an application example of machine learning. Segmentation is, for example, a technique for identifying or detecting a specific region in an image, or for dividing an image into several regions.

[0004] As an application of segmentation in the medical field, a technique for detecting a desired organ or a tumor region from a medical image is known. For example, the segmentation technique can be used to divide a brain region into brain tissue regions such as white matter, gray matter, and cerebrospinal fluid (CSF) in a medical image such as a magnetic resonance (MR) image or a computed tomography (CT) image of the brain. The segmentation technique can also be used to detect a region suspected of being a tumor in a medical image of a patient's brain, or to detect the heart region in a medical image of the chest.

[0005] In machine learning, a trained model is generated by learning using a large amount of training data. For example, when performing image segmentation using machine learning, it is necessary to prepare a large number of real images with annotations for performing the desired segmentation as training data. Annotating a real image means, for example, extracting a specific area in the real image and adding information to the real image indicating what area the extracted area is, or dividing the real image into multiple areas with different properties.

[0006] The process of annotating data (e.g., real images) is almost synonymous with the process of adding correct answer data or labeling, and these processes are usually performed manually. For this reason, it takes a lot of time and cost to prepare a large number of real images (or data with annotations) with annotations as training data. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2022-074088 [Non-patent literature]

[0008] [Non-Patent Document 1] Takeshima H. ​​Deep Learning and Its Application to Function Approximation for MR in Medicine: An Overview. Magn Reson Med Sci doi: 10.2463 / mrms.rev.2121-0040 Summary of the Invention [Problem to be solved by the invention]

[0009] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to generate training data for machine learning, for example, a huge amount of training data for generating a machine learning model for segmentation, at a low cost. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0010] A data processing device of one embodiment includes a first generation unit that generates first artificial data, a second generation unit that generates second artificial data, and a third generation unit that mixes the first artificial data and the second artificial data to generate mixed data and generates mixed region information regarding a region where the first artificial data and the second artificial data are mixed. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the arrangement of a data processing device according to a first embodiment. [Diagram 2] 5 is a flowchart showing an example of processing by the data processing device according to the first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining the processing concept of the data processing device 1 according to the first embodiment. [Figure 4] FIG. 11 is a block diagram showing an example of the arrangement of a data processing device according to a second embodiment. [Diagram 5] 10 is a flowchart showing an example of processing by a data processing device according to the second embodiment. [Figure 6] FIG. 11 is a diagram for explaining the processing concept of a data processing device 1 according to a second embodiment. [Figure 7] FIG. 13 is a block diagram showing an example of the arrangement of a data processing device according to a third embodiment. [Figure 8] 13 is a flowchart showing an example of processing by a data processing device according to the third embodiment. [Figure 9] FIG. 11 is a diagram for explaining the processing concept of a data processing device 1 according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0013] (First embodiment) FIG. 1 is a block diagram showing an example of the configuration of a data processing device 1 according to the first embodiment.

[0014] As shown in FIG. 1, the data processing device 1 of the first embodiment includes a processing circuit 10, a storage circuit 20, an input I / F (interface) 30, and a display 40.

[0015] The processing circuit 10 has a dedicated or general-purpose processor, and realizes various functions described later by software processing by executing a program stored in the storage circuit 20. The processing circuit 10 may be configured with hardware such as an application specific integration circuit (ASIC) or a programmable logic device such as a field programmable gate array (FPGA). The various functions described later can also be realized by hardware processing using these devices. Moreover, the processing circuit 10 may realize various functions described later by combining software processing and hardware processing.

[0016] The input I / F 30 includes an input device that can be operated by a user and an input circuit that inputs a signal from the input device. The input device can be realized by a trackball, a switch, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touchscreen that combines a display screen and a touchpad, a non-contact input device that uses an optical sensor, a voice input device, etc.

[0017] The display 40 is configured by a general display output device such as a liquid crystal display, an OLED (Organic Light Emitting Diode) display, etc. The display 40 cooperates with the input I / F 30 to configure a user interface for inputting and setting various data and information.

[0018] The memory circuitry 20 stores various processing programs used in the processing circuitry 10, data necessary for executing the programs, and various images. The memory circuitry 20 includes a processor-readable storage medium, such as a magnetic or optical storage medium, or a semiconductor memory.

[0019] In addition to the above configuration, the data processing device 1 may include interface circuits for various types of electric communication lines, such as wired / wireless LAN, the Internet, public telephone lines, close-proximity wireless lines such as Bluetooth, etc. Also, the data processing device 1 may include interface circuits for storage media, such as magnetic disks and optical disks, USB memories, various memory cards, etc.

[0020] As shown in FIG. 1, the processing circuit 10 includes a first generating function F01, a second generating function F02, and a third generating function F03.

[0021] The first generation function F01 generates first artificial data, the second generation function F02 generates second artificial data, and the third generation function F03 generates mixed data by mixing the first artificial data and the second artificial data, and generates mixed region information regarding a region where the first artificial data and the second artificial data are mixed.

[0022] Here, as described later, the mixed data and mixed domain information generated by the third generation function F03 are training data for training a machine learning model.

[0023] The data type of the first artificial data is, for example, an image (that is, a first artificial image), but is not limited to an image and may be, for example, sound data, character data, or the like.

[0024] Similarly, the type of the second artificial data is, for example, an image (that is, a second artificial image), but is not limited to an image and may be, for example, sound data, character data, or the like.

[0025] When the first artificial data and the second artificial data are the first artificial image and the second artificial image, respectively, the mixed data generated by the third generation function F03 becomes a mixed image in which the first artificial image and the second artificial image are mixed.

[0026] Moreover, the mixed region information generated by the third generation function F03 is information indicating a concept related to one of the first artificial data and the second artificial data, for example, a region extracted from the second artificial data. When the first artificial data and the second artificial data are the first artificial image and the second artificial image, respectively, the mixed region information is an image expressing a specific region in one of the artificial images, for example, the second artificial image (i.e., a specific region artificial image).

[0027] FIG. 2 is a flowchart showing an example of processing by the data processing device 1 according to the first embodiment, and FIG. 3 is a diagram explaining the processing concept of the data processing device 1 according to the first embodiment.

[0028] 2, a first generating function F01 of the processing circuit 10 generates first artificial data or a first artificial image. Also, in step ST101, a second generating function F02 of the processing circuit 10 generates second artificial data or a second artificial image.

[0029] In step ST102, the third generating function F03 of the processing circuit 10 generates mixed data by mixing the first artificial data and the second artificial data, or generates a mixed image by mixing the first artificial image and the second artificial image.

[0030] In step ST103, the third generating function F03 of the processing circuit 10 generates mixed region information or a specific region artificial image.

[0031] The upper left image in Fig. 3 is a diagram showing an example of a first artificial image as the first artificial data, and the lower left image in Fig. 3 is a diagram showing an example of a second artificial image as the second artificial data. As shown in Fig. 3, the first and second artificial images may be graphic images far removed from real images, or may be artificial images similar to real images. Here, the real image refers to data obtained by imaging the real world using any method, such as a landscape image, a person image, an animal image, or a medical image captured by a medical image diagnostic device such as an MRI device, a CT device, or an ultrasound diagnostic device.

[0032] On the other hand, the first and second artificial images are images artificially generated by artificial intelligence (AI) or machine learning (ML). In other words, the first generating function F01 and the second generating function F02 are generative models (or generators) based on machine learning, and the first and second artificial images are images artificially generated by these generative models.

[0033] As generative models capable of generating the first and second artificial images, several types are known, such as a Generative Adversarial Network (GAN), a Variable Autoencoder (VAE), a Diffusion Model, or an Iterated Function System (IFS).

[0034] It is not necessarily necessary to specify what each of the first artificial image and the second artificial image specifically represents, but it is preferable that at least one of the first artificial image and the second artificial image (for example, in the example of FIG. 3, the second artificial image) is an image representing a specific area.

[0035] For example, the first artificial image may be a background artificial image that simulates the background of the image to be segmented, and the second artificial image may be an artificial image that simulates an image of a segmentation target region (e.g., a star-shaped region) in the image to be segmented.

[0036] In this case, the third generation function F03 generates a mixed image in which the background artificial image and the specific area artificial image are mixed as mixed data, and also generates the specific area artificial image corresponding to the segmentation target area as mixed area information.

[0037] Here, the second artificial image is, for example, an artificial image generated so that the segmentation target region and other regions can be distinguished by transmittance information. Therefore, for example, the second artificial image is generated as an artificial image whose pixel values ​​are the values ​​of the three primary colors R, G, and B and an α value representing transmittance. Then, the segmentation target region can be identified by making the transmittance value different between the segmentation target region and other regions.

[0038] Also, the first artificial image corresponding to the background artificial image can be generated as an artificial image having pixel values ​​of each of the three primary colors R, G, and B and an α value representing transmittance. In this case, for example, a segmentation target region can be identified from the mixed image based on the difference in transmittance between the first artificial image and the second artificial image.

[0039] Alternatively, a second artificial image to which area information corresponding to the segmentation target area is added instead of the transmittance information may be mixed with the first artificial image to generate a mixed image, and the segmentation target area may be identified from the mixed image.

[0040] It is preferable that the first artificial image as a background artificial image and the second artificial image simulating an image of the segmentation target region are generated so as to have different statistical properties. For example, the first artificial image and the second artificial image may be generated so as to have different statistical properties by at least one of the following: (a) using different types of generative models for generating the first and second artificial images, (b) using different configurations or types of generative parameters used in the generative models, and (c) using different pseudo-random number sequences for generation input to the generative models.

[0041] The pair of the mixed image and the specific area artificial image generated by the third generation function F03 in the above manner is used as training data for the first model generation function, which will be described later.

[0042] Second embodiment 4 is a block diagram showing an example of the configuration of a data processing device 1 according to a second embodiment. The data processing device 1 according to the second embodiment has a first trained model that is trained to perform segmentation of a region in the mixed artificial image corresponding to a second artificial image when the mixed artificial image is input. The data processing device 1 also has a function of generating the first trained model by machine learning.

[0043] 4, like the data processing device 1 of the first embodiment, the data processing device 1 of the second embodiment is configured to include a processing circuit 10, a storage circuit 20, an input I / F (interface) 30, and a display 40. However, the second embodiment differs from the first embodiment in the functions realized by the processing circuit 10.

[0044] The processing circuit 10 of the second embodiment has a first model generation function F04 and a first inference function F05 in addition to the first generation function F01, the second generation function F02, and the third generation function F03 of the first embodiment.

[0045] Here, the first generation function F01, the second generation function F02, the third generation function F03, and the first model generation function F04 are functions used in the training phase in machine learning, and the first inference function F05 is a function used in the inference phase.

[0046] Fig. 5 is a flowchart showing an example of processing of the data processing device 1 according to the second embodiment. Fig. 6 is a diagram explaining the processing concept of the data processing device 1 according to the second embodiment. Each of the above functions will be explained below with reference to Figs. 5 and 6.

[0047] The processing from step ST100 to step ST201 in Fig. 5 corresponds to the processing in the training phase. Among these, the processing from step ST100 to step ST103 is the same as that in the first embodiment. In the training phase, a first generating function F01, a second generating function F02, and a third generating function F03 generate a plurality of training data each composed of a pair of mixed data and mixed region information, or a plurality of training data each composed of a pair of a mixed image and a specific region artificial image.

[0048] In step ST200, it is determined whether a predetermined number of training data has been generated. If the predetermined number is not reached, the process from step ST100 to step ST103 is repeated until the predetermined number is reached. If the predetermined number is reached, the process proceeds to step ST201.

[0049] In step ST201, the first generation function F01 generates a first trained model by machine learning using the generated multiple pieces of mixed data and multiple pieces of mixed region information as training data. Alternatively, the first generation function F01 generates a first trained model by machine learning using the generated multiple mixed images and multiple specific region artificial images as training data.

[0050] 6 is a diagram showing the process of step ST201, that is, the process concept of the training phase. Here, the type of the first learning model to be learned is not particularly limited, but is, for example, a machine learning model such as a neural network.

[0051] The multiple mixed data (or multiple mixed images) shown in the upper left of Fig. 6 are, for example, data of training data for which the correct answer is unknown, or input data (or input images) such as images with no annotations or images that have not been segmented. The multiple mixed region information or multiple specific region images shown in the upper right of Fig. 6 are, for example, target region information, or correct data (or correct images) such as images with annotations or images that have been segmented.

[0052] As a result of the learning in step ST201, a first trained model M01 shown in the lower part of FIG. 6 is generated.

[0053] Step ST202 is a process of an inference phase in machine learning, and the lower diagram of Fig. 6 is a diagram showing an operation concept of the inference phase. In the inference phase, mixed data (or a mixed image) different from that in the learning phase is input to the first trained model, and mixed region information (or a specific region artificial image) is acquired from the first trained model. For example, by inputting a mixed artificial image as shown in the lower left diagram of Fig. 6 to the first trained model, a specific region artificial image in which a specific region is segmented, for example, as shown in the lower right diagram of Fig. 6, is output from the first trained model.

[0054] (Third embodiment) 7 is a block diagram showing a configuration example of a data processing device 1 according to a third embodiment. The data processing device 1 of the third embodiment generates a second trained model by performing transfer learning on a first trained model generated by the data processing device 1 of the second embodiment.

[0055] As shown in FIG. 7, the data processing device 1 of the third embodiment is configured to include a processing circuit 10, a memory circuit 20, an input I / F (interface) 30, and a display 40, similar to the data processing device 1 of the second embodiment.

[0056] However, the processing circuit 10 of the third embodiment has a second model generation function F06 and a second inference function F07 in addition to the first generation function F01, the second generation function F02, the third generation function F03, and the first model generation function F04 that are possessed in the second embodiment.

[0057] Fig. 8 is a flowchart showing an example of processing of the data processing device 1 according to the third embodiment. Fig. 9 is a diagram explaining the processing concept of the data processing device 1 according to the second embodiment. Each of the above functions will be explained below with reference to Figs. 8 and 9.

[0058] The processes from step ST100 to step ST201 in Fig. 8 are the same as those in the second embodiment, and therefore description thereof will be omitted. Note that the processes from step ST100 to step ST201 are processes related to the training phase in the second embodiment. These processes are processes up to generating a first trained model M01 using a plurality of training data consisting of a plurality of mixed data (or a plurality of mixed artificial images) and a plurality of mixed region information (or a plurality of specific region artificial images) corresponding to the plurality of mixed data, as shown in the upper part of Fig. 6.

[0059] In step ST300, the second model generation function F06 generates the second trained model M02 by transfer learning using a predetermined number of real images and annotation information attached to each of the real images as training data. Alternatively, in step ST300, the second model generation function F06 generates the second trained model M02 by transfer learning using a plurality of medical images and segmentation information attached to each of the medical images as training data.

[0060] Here, transfer learning is, for example, a technique for applying knowledge from one domain to learning in another domain, or a machine learning technique that uses a model trained for one task (trained model) as the starting point for a model that performs another task.

[0061] The middle diagram in Figure 9 illustrates the concept of the process of performing transfer learning on a first trained model M01 generated based on artificial data or an artificial image, using multiple medical images that are real images and multiple medical images each having annotations added thereto, to generate a second trained model M02.

[0062] The medical images used as input data in transfer learning are images of actual patients captured by medical imaging diagnostic devices such as an MRI device, an X-ray CT device, or an ultrasound diagnostic device. The annotated medical images used as correct answer data in transfer learning are images that a user such as a doctor reads the medical images as input data and adds annotation information such as the disease site (e.g., a tumor) and the name of the disease to the medical images.

[0063] 8 is a process of an inference phase using the second trained model M02 generated by transfer learning. In the inference phase, an actual image (or a medical image) different from the image used in the transfer learning phase is input to the second trained model M02 to obtain annotation information (or segmentation information), or to obtain an image (or a segmented image) to which annotation information is added.

[0064] The lower diagram in Fig. 9 illustrates the processing concept of the inference phase. In this example, for example, a medical image (e.g., an MR image) of a patient's head is input to the second trained model M02, and an annotated medical image, for example, a medical image annotated with a message indicating that there is a suspected brain tumor and the location of the tumor, is output.

[0065] According to the data processing device 1 according to the third embodiment described above, the first trained model M01 is trained using artificial data (or artificial images) generated by a machine learning model such as GAN, VAE, Diffusion Model, IFS, etc., as training data. Therefore, there is no particular limit to the number of training data, and any desired number of training data can be generated. For example, it is easy to generate one million pieces of training data.

[0066] On the other hand, in the transfer learning phase, for example, the training data are medical images of actual patients and annotation information added to these medical images as the diagnosis results of specialists such as doctors. For this reason, creating training data requires cost and time, and as a result, the number of training data is limited.

[0067] In general, when a trained model is created from scratch, if the number of training data is small, the accuracy of the trained model is low, and the probability of deriving a correct answer is reduced. However, in the data processing device 1 of the third embodiment, the second trained model M02 to be finally used is generated by transfer learning to the first trained model M01 trained by a huge number of artificial data. Therefore, even if the number of actual medical images and annotated medical images used for transfer learning is small, for example, about 1000, it is possible to improve the accuracy of the second trained model M02 to be finally used, and it is possible to derive a highly reliable result from the second trained model M02.

[0068] The first generation function, second generation function, third generation function, first model generation function, and second model generation function in the above descriptions of the data processing device 1 according to each embodiment are examples of the first generation unit, second generation unit, third generation unit, first model generation unit, and second model generation unit, respectively, as described in the claims.

[0069] According to the data processing device of each embodiment described above, training data for machine learning, for example, a huge amount of training data for generating a machine learning model for segmentation, can be generated at low cost.

[0070] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0071] 1 Data Processing Device 10 Processing circuit 20 Memory circuit 30 Input I / F 40 Display F01 First generation function F02 2nd generation function F03 Third generation function F04 First model generation function F05 First inference function F06 Second model generation function F07 Second inference function

Claims

1. A first generation unit that generates first artificial data; A second generation unit that generates second artificial data; a third generation unit that generates mixed data by mixing the first artificial data and the second artificial data, and generates mixed region information regarding a region in which the first artificial data and the second artificial data are mixed; A data processing device comprising:

2. The mixed data and the mixed region information are training data for learning a machine learning model.

2. A data processing apparatus according to claim 1.

3. The first artificial data is a first artificial image, the second artificial data is a second artificial image, and the mixed data is a mixed artificial image obtained by mixing the first artificial image and the second artificial image.

2. A data processing apparatus according to claim 1.

4. the first generation unit and the second generation unit are generative models based on machine learning capable of generating artificial data or artificial images, and the first generation unit and the second generation unit differ in at least one of (a) the type of the generative model, (b) a generation parameter also used inside the generative model, and (c) a pseudo-random number sequence for generation input to the generative model; 2. A data processing apparatus according to claim 1.

5. the first artificial image generated by the first generation unit is a background artificial image that simulates a background of an image to be segmented, the second artificial image generated by the second generation unit is an artificial image simulating an image of a segmentation target region in the segmentation target image, the third generating unit generates a mixed artificial image in which the background artificial image and the simulated artificial image are mixed as the mixed data, and generates a specific area artificial image corresponding to the segmentation target area as the mixed area information.

4. A data processing device according to claim 3.

6. The second artificial image is a specific area artificial image generated so that the segmentation target area and other areas can be distinguished from each other based on transmittance information.

6. A data processing apparatus according to claim 5.

7. the third generation unit uses region information corresponding to the first artificial image, the second artificial image, and a segmentation target region defined for the second artificial image to generate the mixed artificial image by mixing the first artificial image and the second artificial image to which the region information has been added.

6. A data processing apparatus according to claim 5.

8. the background artificial image and the artificial image simulating the image of the segmentation target region are generated so as to have different statistical properties from each other; 6. A data processing apparatus according to claim 5.

9. a first trained model trained to segment a region in the mixed artificial image corresponding to the second artificial image when the mixed artificial image is input; The data processing apparatus of claim 3 further comprising:

10. A first model generation unit that generates a first trained model, The first model generation unit generates the first trained model by machine learning using the plurality of pieces of mixed data and the plurality of pieces of mixed region information generated by the third generation unit as training data.

2. A data processing apparatus according to claim 1.

11. A first model generation unit that generates a first trained model, the first model generation unit generates the first trained model by machine learning using the plurality of background artificial images and the plurality of specific area images generated by the third generation unit as training data; 6. A data processing apparatus according to claim 5.

12. A second model generation unit that performs transfer learning on the first trained model to generate a second trained model, The second model generation unit generates the second trained model by transfer learning using a plurality of real images and annotation information assigned to each of the plurality of real images as training data.

11. A data processing apparatus according to claim 10.

13. A second model generation unit that performs transfer learning on the first trained model to generate a second trained model, The second model generation unit generates the second trained model by transfer learning using a plurality of medical images and segmentation information assigned to each of the plurality of medical images as training data. A data processing apparatus according to claim 11.

14. The first generation unit and the second generation unit are machine learning models capable of generating artificial images, including at least one of a Generative Adversarial Network (GAN), a Variable Autoencoder (VAE), a Diffusion Model, and an Iterated Function System (IFS), 2. A data processing apparatus according to claim 1.

15. A data processing device according to any one of claims 1 to 14. A medical image processing device comprising:

16. Generate first artificial data; Generate second artificial data; mixing the first artificial data and the second artificial data to generate mixed data, and generating mixed region information relating to a region in which the first artificial data and the second artificial data are mixed; Data processing methods.

Images