Learning data generation device, learning processing device, image data generation device, learning data generation method, learning processing method, image data generation method, and program

Abstract

Provided is a medical diagnosis learning data generation system capable of: collecting a large amount of highly accurate learning data (medical image data (for example, medical diagnostic image data)) required for acquiring a high-performance trained model by machine learning; and performing accurate annotation. In a medical diagnostic learning data generation system (1000), various learning image data can be acquired by a learning data generation device (100). In the medical diagnosis learning data generation system (1000), a learning processing device (200) performs learning processing using the various learning image data acquired by the learning data generation device (100) to acquire a trained model. Then, the trained model is deployed on a medical image data generation device (300), label input is performed, and inference processing is performed, whereby a large amount of highly accurate labeled medical image data can be generated.

Description

Learning data generation device, learning processing device, image data generation device, learning data generation method, learning processing method, image data generation method, and program

[0001] The present invention relates to a technology for acquiring a large amount of highly accurate learning data to be used for learning to generate a medical AI model.

[0002] In recent years, a technology has been developed in which medical images (e.g., X-ray images) are input into a trained model obtained through machine learning, and a diagnostic result is output from the trained model (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2018-32071

[0004] In the conventional techniques described above, machine learning must be performed using medical diagnostic images, but in order to obtain a high-performance trained model through this machine learning, a large amount of highly accurate training data (medical diagnostic images) is required.

[0005] However, it is difficult to collect a large amount of highly accurate learning data (medical diagnostic images (medical data)) for the following reasons.

[0006] Medical data is individualized: each patient has their own unique symptoms and constitution, and there is a lack of sufficient data to cover this diversity.

[0007] Furthermore, medical data is extremely sensitive and requires strict security measures when it is handled, making it difficult to collect large amounts of medical data.

[0008] Additionally, medical data presents challenges in terms of data sharing, namely the complexity of sharing medical data while protecting patient privacy, which is a barrier to data collection.

[0009] Furthermore, it is difficult to inexpensively annotate training data (medical diagnostic images (medical data)). In other words, annotating medical data (medical diagnostic images) requires highly specialized knowledge and expert judgment, which results in time-consuming and costly annotation of training data (medical diagnostic images (medical data)).

[0010] As such, it is difficult to collect large amounts of highly accurate training data (medical diagnostic images) and perform accurate annotation, which are necessary to obtain high-performance trained models through machine learning.

[0011] In view of the above problems, the present invention aims to provide a medical diagnostic training data generation system, a training data generation device, a training processing device, an image data generation device, a training data generation method, a training processing method, an image data generation method, and a program that can collect large amounts of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning and perform accurate annotation.

[0012] In order to solve the above problem, a first invention is a training data generation device comprising an image input unit, a standardization unit, a feature extraction image generation unit, a first editing processing unit, a second editing processing unit, a manually generated image input unit, and a training data generation unit.

[0013] The image input unit inputs image data.

[0014] The standardization unit performs standardization processing on the image data input to the image input unit, thereby obtaining standardized image data.

[0015] The feature extraction image generation unit performs feature extraction processing on the standardized image data to obtain feature extraction image data.

[0016] The first editing processing unit can input first editing data, which is data for editing feature-extracted image data, and performs editing processing on the feature-extracted image data based on the first editing data, and obtains the image data after the editing processing as first edited image data.

[0017] The second editing processing unit can input second editing data, which is data for editing the standardized image data, and performs editing processing on the standardized image data based on the second editing data, and acquires the image data after the editing processing as the second edited image data. The manually generated image input unit is a functional unit for inputting manually generated image data, which is data for a manually generated map image.

[0018] The learning data generation unit acquires learning data that is a combination of standardized image data and one of the following image data: (1) feature-extracted image data, (2) first edited image data, (3) first edited image data, and (4) manually generated image data.

[0019] With the above configuration, this training data generation device can acquire a variety of training image data, namely: (1) training data that combines standardized image data (actual image data) and map image (condition image) data (image data from which features have been extracted) automatically generated from the actual image data; (2) training data that combines standardized image data (actual image data) and map image (condition image) data obtained by editing a map image automatically generated from the actual image data; (3) training data that combines standardized image data (actual image data) and map image (condition image) data obtained by performing an editing process based on the actual image data; and (4) training data that combines standardized image data (actual image data) and manually generated map image (condition image) data. Then, by using the various training image data acquired by this training data generation device to train a trainable model (e.g., a conditional GAN ​​model), highly accurate training processing can be performed. Then, by performing inference processing using the trained model obtained through the learning process, it is possible to obtain a large amount of highly accurate generated image data (e.g., medical image data) even when only a small amount of actual image data can be obtained.

[0020] Note that "standardization processing" refers to a process in which, for example, an input image is subjected to extraction processing of a specified image area, padding processing of the specified image area, and image resizing processing to obtain standardized image data.

[0021] A second invention is the learning processing device according to the first invention, comprising a data input unit, a learning processing unit, and a loss evaluation unit.

[0022] The data input unit inputs the learning data generated by the learning data generating device of the first invention, and extracts condition image data and actual image data from the learning data.

[0023] The learning processing unit includes a trainable model, receives the condition image data and the actual image data, and performs a learning process for the trainable model.

[0024] The loss evaluation unit evaluates the error between the data output from the learning processing unit and the training data.

[0025] As a result, this learning processing device can perform a learning process for a trainable model using the learning data generated by the learning data generation device of the first invention. In other words, even if only a small amount of actual image data can be acquired, the learning processing can be efficiently performed to obtain a trained model that can acquire a large amount of highly accurate generated image data (e.g., medical image data).

[0026] A third aspect of the present invention is the second aspect of the present invention, wherein the learning processing unit includes a generator and a classifier.

[0027] The generator has a trainable model that receives a condition image as input and generates image data based on the input condition image.

[0028] The classifier inputs the condition image and either the image data generated by the generator or the actual image data, and outputs the result of the determination process as to whether the input data is genuine data or fake data.

[0029] As a result, this learning processing device can perform learning processing by adopting, for example, a conditional GAN ​​model as a trainable model.

[0030] In addition, the method of inputting the condition image and either the image data generated by the generator or the actual image data may be, for example, a method of inputting data that is a concatenation of the condition image and the image data generated by the generator, or a method of inputting data that is a concatenation of the condition image and the actual image data.

[0031] Furthermore, there may be a plurality of classifiers, and data of different scales (for example, downsampled data) may be input to the plurality of classifiers.

[0032] A fourth invention is an image data generation device comprising an image input unit, a standardization unit, a feature extraction image generation unit, a first editing processing unit, a second editing processing unit, a manually generated image input unit, a map image generation unit, an inference generator, a label input unit, and a generated data acquisition processing unit.

[0033] The image input unit inputs image data.

[0034] The standardization unit performs standardization processing on the image data input to the image input unit, thereby obtaining standardized image data.

[0035] The feature extraction image generation unit performs feature extraction processing on the standardized image data to obtain feature extraction image data.

[0036] The first editing processing unit can input first editing data, which is data for editing feature-extracted image data, and performs editing processing on the feature-extracted image data based on the first editing image, and obtains the image data after the editing processing as the first editing image data.

[0037] The second editing processing unit can input second editing data, which is data for editing the standardized image data, and performs editing processing on the standardized image data based on the second editing image, and obtains the image data after the editing processing as the second editing image data.

[0038] The manually generated image input unit inputs manually generated image data, which is data for a manually generated map image.

[0039] The map image generation unit acquires one of the following image data as map image data: (1) feature extraction image data, (2) first edited image data, (3) first edited image data, and (4) manually generated image data.

[0040] The inference generator is equipped with a trained model of a trainable model obtained by a learning process by the learning processing device described in the second invention, or a trained model of a trainable model of the generator of the learning processing device described in the third invention obtained by a learning process by the learning processing device described in the third invention, and generates inference processing result image data from map image data.

[0041] The label input unit inputs a label for the map image data.

[0042] The generated data acquisition processing unit acquires labeled image data that is a combination of the label input to the label input unit and the inference processing result image data generated by the inference generator.

[0043] With the above configuration, this image data generation device can acquire a variety of input image data (input data for inference processing) based on a small number of real image data, i.e., (1) map image (condition image) data (image data from which features have been extracted) automatically generated from standardized real image data, (2) map image (condition image) data acquired by editing a map image automatically generated from standardized real image data, (3) map image (condition image) data acquired by performing editing processing based on the standardized real image data, and (4) manually generated map image (condition image) data.The image data generation device can acquire highly accurate generated image data by inputting the various input data for inference (image data) acquired as described above into an inference generator equipped with a trainable model. Furthermore, in the image data generation device, the user knows in advance what kind of data the image data (actual image data) input to the second actual image input unit 30 or the data input to the manually-generated image input unit is (the user knows in advance that the actual image data input to the image input unit or the data input to the manually-generated image input unit is, for example, medical image data of a subject suffering from a specific disease or medical image data of a healthy state), so data (information) for identifying the label corresponding to the map image output from the map image generation unit can be accurately input to the label input unit.The generated data acquisition processing unit of this image data generation device can then acquire data (paired data (labeled data)) that pairs the generated image data output from the inference generator with the label data output from the label input unit, making it possible to easily acquire large amounts of accurately annotated data (labeled data).

[0044] Therefore, this image data generation device can collect a large amount of high-precision training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation.A fifth invention is a training data generation method comprising an image input step, a standardization step, a feature extraction image generation step, a first editing processing step, a second editing processing step, a manual image generation step, and a training data generation step.

[0045] The image input step inputs image data.

[0046] In the standardization step, standardized image data is obtained by performing a standardization process on the image data input in the image input step.

[0047] The feature-extracted image generating step performs feature extraction processing on the standardized image data to obtain feature-extracted image data.

[0048] The first editing processing step can input first editing data, which is data for editing the feature-extracted image data, and perform editing processing on the feature-extracted image data based on the first editing data, and obtain the image data after the editing processing as the first edited image data. The second editing processing step can input second editing data, which is data for editing the standardized image data, and perform editing processing on the standardized image data based on the second editing data, and obtain the image data after the editing processing as the second edited image data.

[0049] The manually generated image input step inputs manually generated image data, which is data for a manually generated map image.

[0050] The learning data generation step acquires learning data that is a combination of standardized image data and one of the following image data: (1) feature-extracted image data, (2) first edited image data, (3) first edited image data, and (4) manually generated image data.

[0051] This makes it possible to realize a learning data generation method that has the same effects as the first aspect of the invention.

[0052] A sixth aspect of the present invention is a learning processing method for performing learning processing using training data generated by the training data generation method of the fifth aspect of the present invention, the learning processing method including a data input step, a learning processing step, and a loss evaluation step. The data input step inputs training data generated by the training data generation method according to the fifth aspect of the present invention, and extracts condition image data and actual image data from the training data. The learning processing step inputs the condition image data and the actual image data, and performs learning processing of a trainable model. The loss evaluation step evaluates the error between the data acquired by the learning processing step and training data.

[0053] This makes it possible to realize a learning processing method that has the same effect as the second aspect of the invention.

[0054] A seventh aspect of the present invention is the sixth aspect of the present invention, wherein the learning process step includes a generating step and an identifying step.

[0055] The generation step inputs a condition image and generates image data using a trainable model based on the input condition image.

[0056] The identification step inputs a condition image and either the image data generated by the generation step or the actual image data, and outputs the result of the determination process as to whether the input data is genuine data or fake data.

[0057] This makes it possible to realize a learning processing method that has the same effect as the third aspect of the invention.

[0058] The eighth invention is an image data generation method including an image input step, a standardization step, a feature extraction image generation step, a first editing processing step, a second editing processing step, a manually generated image input step, a map image generation step, an inference generation step, a label input unit, and a generated data acquisition processing step.

[0059] The image input step inputs image data.

[0060] The standardization step performs standardization processing on the image data input in the image input step to obtain standardized image data.

[0061] The feature-extracted image generating step performs feature extraction processing on the standardized image data to obtain feature-extracted image data.

[0062] The first editing processing step can input first editing data, which is data for editing the feature-extracted image data, and perform editing processing on the feature-extracted image data based on the first editing image, and obtain the image data after the editing processing as the first edited image data. The second editing processing step can input second editing data, which is data for editing the standardized image data, and perform editing processing on the standardized image data based on the second editing image, and obtain the image data after the editing processing as the second edited image data.

[0063] The manually generated image input step inputs manually generated image data, which is data for a manually generated map image.

[0064] The map image generation step acquires one of the following image data as map image data: (1) feature-extracted image data, (2) first edited image data, (3) second edited image data, and (4) manually generated image data.

[0065] The inference generation step generates inference processing result image data from map image data using a trained model of a trainable model obtained by a learning process using the learning processing method described in the sixth invention, or a trained model of a trainable model used in the generation step of the learning processing method described in the seventh invention obtained by a learning process using the learning processing method described in the seventh invention.

[0066] The label input unit inputs a label for the map image data.

[0067] The generated data acquisition processing step acquires labeled image data that is a combination of the label input to the label input section and the inference processing result image data generated in the inference generation step.

[0068] This makes it possible to realize an image data generating method that has the same effect as the fourth aspect of the invention.

[0069] A ninth aspect of the present invention is a program for causing a computer to execute the learning data generating method of the fifth aspect of the present invention.

[0070] This makes it possible to realize a program for causing a computer to execute a learning data generation method that has the same effect as the fifth aspect of the invention.

[0071] A tenth aspect of the present invention is a program for causing a computer to execute the learning processing method according to the sixth or seventh aspect of the present invention.

[0072] This makes it possible to realize a program for causing a computer to execute a learning processing method that has the same effects as the sixth or seventh aspect of the invention.

[0073] An eleventh aspect of the present invention is a program for causing a computer to execute the image data generating method of the eighth aspect of the present invention.

[0074] This makes it possible to realize a program for causing a computer to execute an image data generating method that has the same effect as the eighth aspect of the invention.

[0075] According to the present invention, it is possible to realize a medical diagnostic training data generation system, a training data generation device, a training processing device, an image data generation device, a training data generation method, a training processing method, an image data generation method, and a program that can collect large amounts of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning and perform accurate annotation.

[0076] FIG. 1 is a diagram showing a schematic configuration of a medical diagnostic training data generation system 1000 according to the first embodiment. FIG. 2 is a diagram showing a schematic configuration of a training data generation device 100 of the medical diagnostic training data generation system 1000 according to the first embodiment. FIG. 3 is a diagram showing a schematic configuration of a training processing device 200 of the medical diagnostic training data generation system 1000 according to the first embodiment. FIG. 4 is a diagram showing a schematic configuration of a medical image data generation device 300 of the medical diagnostic training data generation system 1000 according to the first embodiment. FIG. 5 is a diagram for explaining the processing (standardization processing) of a standardization unit 12. FIG. 6 is a diagram for explaining the processing (standardization processing) of the standardization unit 12. FIG. 7 is a diagram for explaining the processing (feature extraction processing) of a feature extraction image generation unit 13. FIG. 8 is a diagram for explaining the processing (first editing processing) of a first editing processing unit 14. FIG. 9 is a diagram for explaining the processing (second editing processing) of a second editing processing unit 15. FIG. 10 is a diagram showing a CPU bus configuration.

[0077] First Embodiment A first embodiment will be described below with reference to the drawings.

[0078] <1.1: Configuration of Medical Diagnosis Training Data Generation System> FIG. 1 is a diagram showing the schematic configuration of a medical diagnosis training data generation system 1000 according to the first embodiment.

[0079] FIG. 2 is a diagram showing a schematic configuration of the training data generation device 100 of the medical diagnosis training data generation system 1000 according to the first embodiment.

[0080] FIG. 3 is a diagram showing a schematic configuration of the learning processing device 200 of the medical diagnosis learning data generation system 1000 according to the first embodiment.

[0081] FIG. 4 is a diagram showing a schematic configuration of the medical image data generating device 300 of the medical diagnostic training data generating system 1000 according to the first embodiment.

[0082] As shown in FIG. 1, the medical diagnosis training data generation system 1000 includes a training data generation device 100, a first data storage unit DB1, a training processing device 200, a medical image data generation device 300, and a second data storage unit DB2.

[0083] (1.1.1: Training data generation device 100) As shown in FIG. 2 , the training data generation device 100 includes an actual image input unit 11, a standardization unit 12, a feature extraction image generation unit 13, a first editing processing unit 14, a second editing processing unit 15, a manually generated image input unit 16, and a training data generation unit 17.

[0084] The real image input unit 11 is a functional unit (e.g., a data input interface) for inputting actual medical image data (real medical image data). The real image input unit 11 outputs data including the input medical image data to the standardization unit 12 as data Din_real.

[0085] The standardization unit 12 inputs the data Din_real output from the real image input unit 11, acquires the medical image data (data of the real medical image) included in the data Din_real, and performs a standardization process on the medical image data (data of the real medical image).The standardization unit 12 then outputs data including the data (image data) after the standardization process as data D1_real to the feature extraction image generation unit 13, the second editing processing unit 15, and the learning data generation unit 17.

[0086] The feature extraction image generation unit 13 inputs the data D1_real output from the standardization unit 12, performs feature extraction image generation processing on the standardized image data included in the data D1_real, and generates (acquires) a feature extraction image through this processing.The feature extraction image generation unit 13 then outputs data including the feature extraction image generated (acquired) through the feature extraction image generation processing as data D1_map_auto to the first edit processing unit 14 and the learning data generation unit 17.

[0087] The first editing processing unit 14 receives data D1_map_auto output from the feature extraction image generation unit 13 and data D11_stylus indicating the editing content (the data D11_stylus is, for example, data input from an external device). The first editing processing unit 14 performs editing processing on the feature extraction image included in the data D1_map_auto based on the data D11_stylus indicating the editing content, and obtains image data after the editing processing. The first editing processing unit 14 then outputs data including the image data after the editing processing as data D11_map_edited to the learning data generation unit 17.

[0088] The second editing processing unit 15 receives the data D1_real output from the standardization unit 12 and data D12_stylus, which indicates the editing content (the data D12_stylus is, for example, data input from an external source). Based on the data D12_stylus, which indicates the editing content, the second editing processing unit 15 performs editing on the standardized image (image data) included in the data D1_real, and obtains the image data after the editing process. The second editing processing unit 15 then outputs data including the image data after the editing process to the learning data generation unit 17 as data D12_map_edited.

[0089] The manually generated image input unit 16 is a functional unit (e.g., a data input interface) for inputting manually generated image data. The manually generated image input unit 16 outputs data including the input image data to the learning data generation unit 17 as data D1_map_manual.

[0090] The training data generation unit 17 receives as input the data D1_real output from the standardization unit 12, the data D1_map_auto output from the feature extraction image generation unit 13, the data D11_map_edited output from the first editing processing unit 14, the data D12_map_edited output from the second editing processing unit 15, and the data D1_map_manual output from the manually generated image input unit 16. The training data generation unit 17 also receives as input a mode signal Mode. The mode signal Mode is generated by a control unit (not shown) that controls the functional units of the training data generation device 100 and is output from the control unit to the training data generation unit 17.

[0091] In accordance with the mode signal Mode, the training data generation unit 17 pairs (1) the data D1_real output from the standardization unit 12 with one of (2A) the data D1_map_auto output from the feature extraction image generation unit 13, (2B) the data D11_map_edited output from the first editing processing unit 14, (2C) the data D12_map_edited output from the second editing processing unit 15, and (2D) the data D1_map_manual output from the manually generated image input unit 16, thereby acquiring training data (training data in which standardized real image data and map image data (condition image data) are paired).The training data generation unit 17 then outputs the acquired training data as data Dtrain_pair (for example, to the first data storage unit DB1).

[0092] (1.1.2: First Data Storage Unit DB1) The first data storage unit DB1 is a functional unit that can read data from a predetermined area (memory area) of the first data storage unit DB1 in accordance with an external data read command and / or write data to a predetermined area (memory area) of the first data storage unit DB1 in accordance with an external data write command. The first data storage unit DB1 is realized by, for example, a data storage device, a database, or a memory device.

[0093] In accordance with a data write command from the learning data generation device 100, the first data memory unit DB1 writes the data Dtrain_pair output from the learning data generation unit 17 of the learning data generation device 100 into a specified area (memory area) of the first data memory unit DB1.

[0094] In addition, the first data storage unit DB1 reads data stored in a specified area (memory area) of the first data storage unit DB1 in accordance with a data read command from the learning processing device 200, and outputs the read data to the learning processing device 200.

[0095] 3 , the learning processing device 200 includes a data input unit 21, a generator 22, a first combination unit 23A, a second combination unit 23B, a selector SEL1, a downsampling unit 24, a first classifier 25A, a second classifier 25B, and a loss evaluation unit 26. The learning processing device 200 is a device for executing a learning process using a generative adversarial network model (GAN model (GAN: Generative Adversarial Network)) having a conditional input, a so-called conditional GAN ​​model (trainable model).

[0096] The data input unit 21 outputs a data read command to the first data storage unit DB1, reads out learning data (learning data in which standardized actual image data and map image data (condition image data) are paired) stored in a predetermined area (memory area) of the first data storage unit DB1, and inputs the read learning data as data Din_pair (learning data Din_pair).The data input unit 21 extracts map image data (condition image data) from the learning data Din_pair, and outputs the extracted map image data (condition image data) as data Din_condition to the generator 22, the first combination unit 23A, and the second combination unit 23B.

[0097] Furthermore, the data input unit 21 extracts standardized real image data from the learning data Din_pair, and outputs the extracted standardized real image data to the second combining unit 23B as data Din_real.

[0098] The generator 22 is a generator of a conditional GAN ​​model, and is a trainable model (the generator parameters are θ （Ｇ） The generator 22 inputs the map image data (condition image data) Din_condition output from the data input unit 21. The generator 22 inputs the map image data (condition image data) Din_condition to the trainable model of the generator 22, executes processing using the trainable model, and outputs the data acquired by the processing to the first combination unit 23A as data Do_G.

[0099] The first combination unit 23A inputs the data Do_G output from the generator 22 and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Do_G and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sG.

[0100] The second combining unit 23B inputs the data Din_real and data Din_condition output from the data input unit 21, performs a process of concatenating the data Din_real and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sx.

[0101] The selector SEL1 is a two-input, one-output selector that selects either the data Do_G output from the first combination unit 23A or the data Do_sx output from the second combination unit 23B in accordance with a selection signal sel1 from a control unit (not shown) that controls the functional units of the learning processing device 200, and outputs the selected data as data Do0 to the downsampling unit 24 and the second classifier 25B.

[0102] The downsampling unit 24 inputs data Do0 output from the selector SEL1, performs a downsampling process on the data Do0, obtains the downsampled data, and outputs the obtained data as data Do1 to the first identifier 25A.

[0103] The first discriminator 25A is a discriminator of a conditional GAN ​​model, and is a trainable model (the parameters of the first discriminator are set as θ （Ｄ１） and includes T (T: natural number) layers (layers that extract features). The first classifier 25A receives the data Do1 output from the downsampling unit 24. The first classifier 25A receives the data Do1 output from the downsampling unit 24 as input to the trainable model of the first classifier 25A, executes processing using the trainable model (processing for determining whether the input data Do1 is genuine or fake), and outputs data including determination result data obtained by the processing and output data (feature data) of the T (T: natural number) layers (layers that extract features) to the loss evaluation unit 26 as data Do_D1.

[0104] The second discriminator 25B is a discriminator of a conditional GAN ​​model, and is a trainable model (the parameters of the second discriminator are set as θ （Ｄ２） The second classifier 25B receives the data Do0 output from the selector SEL1 as input. The second classifier 25B executes processing using the trainable model (processing for determining whether the input data Do0 is genuine or fake), and outputs data including determination result data acquired by the processing and output data (feature data) of the T layers (layers for extracting features) (T: a natural number) as data Do_D2 to the loss evaluation unit 26.

[0105] The loss evaluation unit 26 receives the determination result data Do_D1 output from the first classifier 25A, the determination result data Do_D2 output from the second classifier 25B, and the teacher data D_correct, and obtains an error (loss obtained by a loss function) based on the determination result data Do_D1 and the teacher data D_correct, and an error (loss obtained by a loss function) based on the determination result data Do_D2 and the teacher data D_correct. Then, the loss evaluation unit 26 performs parameter update processing (learning processing) of the trainable models of the first classifier 25A, the second classifier 25B, and the generator 22 based on the obtained errors (losses) (for example, performing learning processing using an error backpropagation method). In addition, the teacher label (corresponding to the teacher data D_correct) for the judgment data as to whether it is genuine or fake, contained in the output data Dout_D1 and Dout_D2 of the trainable models of the first classifier 25A and the second classifier 25B, is “1” indicating that it is genuine data when data including real data (including data after downsampling) is input to the first classifier 25A and the second classifier 25B, and is “0” indicating that it is fake data when data including fake data (output data of the generator 22) (including data after downsampling) is input to the first classifier 25A and the second classifier 25B.

[0106] Furthermore, the loss evaluation unit 26 inputs the data Din_condition (the probability distribution of this data is assumed to be "s") output from the data input unit 21, the data Do_G (the probability distribution of this data is assumed to be "G(s)") output from the generator 22, and the data Din_real (the probability distribution of this data is assumed to be "x") output from the data input unit 21, and evaluates the loss of the conditional GAN ​​model using a loss function based on the input data, and performs a process of determining whether or not the learning process has converged.

[0107] (1.1.4: Medical image data generation device 300) As shown in Figure 4, the medical image data generation device 300 includes a second actual image input unit 30, a second standardization unit 31, a second feature extraction image generation unit 32, a third editing processing unit 33, a fourth editing processing unit 34, a second manually generated image input unit 35, a map image generation unit 36, a label input unit 37, a generator 38, and a generated data acquisition processing unit 39.

[0108] The second real image input unit 30 is a functional unit (e.g., a data input interface) for inputting actual medical image data (real medical image data). The second real image input unit 30 outputs data including the input medical image data to the second standardization unit 31 as data Din2_real.

[0109] The second standardization unit 31 inputs the data Din2_real output from the second real image input unit 30, acquires the medical image data (data of the real medical image) included in the data Din2_real, and performs a standardization process on the medical image data (data of the real medical image).The second standardization unit 31 then outputs data including the data (image data) after the standardization process as data D2_real to the second feature extraction image generation unit 32 and the third editing processing unit 33.

[0110] The second feature extraction image generation unit 32 receives the data D2_real output from the second standardization unit 31, performs a feature extraction image generation process on the standardized image data included in the data D2_real, and generates (acquires) a feature extraction image through this process.The second feature extraction image generation unit 32 then outputs data including the feature extraction image generated (acquired) through the feature extraction image generation process to the third editing processing unit 33 as data D2_map_auto.

[0111] The third edit processing unit 33 receives data D2_map_auto output from the second feature extraction image generation unit 32 and data D21_stylus (data D21_stylus is, for example, data input from an external source) that indicates the editing content. Based on the data D21_stylus that indicates the editing content, the third edit processing unit 33 performs an edit process on the feature extraction image included in data D21_map_auto and obtains image data after the edit process. The third edit processing unit 33 then outputs data including the image data after the edit process to the map image generation unit 36 ​​as data D21_map_edited.

[0112] The fourth editing processing unit 34 receives the data D2_real output from the second standardization unit 31 and data D22_stylus (for example, data input from an external source) that indicates the editing content. Based on the data D22_stylus that indicates the editing content, the fourth editing processing unit 34 performs editing on the standardized image (image data) included in the data D2_real, and obtains the image data after the editing process. The fourth editing processing unit 34 then outputs data including the image data after the editing process to the map image generation unit 36 ​​as data D22_map_edited.

[0113] Second manually generated image input unit 35 is a functional unit (e.g., a data input interface) for inputting manually generated image data. Second manually generated image input unit 35 outputs data including the input image data to map image generation unit 36 ​​as data D2_map_manual.

[0114] The map image generation unit 36 ​​receives as input the data D2_map_auto output from the second feature extraction image generation unit 32, the data D21_map_edited output from the third edit processing unit 33, the data D22_map_edited output from the fourth edit processing unit 34, and the data D2_map_manual output from the second manually generated image input unit 35. The map image generation unit 36 ​​also receives as input a mode signal Mode. The mode signal Mode is generated by a control unit (not shown) that controls the functional units of the medical image data generation device 300 and is output from the control unit to the map image generation unit 36.

[0115] In accordance with the mode signal Mode, the map image generation unit 36 ​​selects one of (1) data D2_map_auto output from the second feature extraction image generation unit 32, (2) data D21_map_edited output from the third editing processing unit 33, (3) data D22_map_edited output from the fourth editing processing unit 34, and (4) data D2_map_manual output from the second manually generated image input unit 35, and outputs the selected data to the generator 38 as data Do_map.

[0116] The label input unit 37 is a functional unit (e.g., a data input interface) for inputting data (information) for identifying a label corresponding to the map image output from the map image generation unit 36. The label input unit 37 outputs the input data (label data) to the generated data acquisition processing unit 39 as data Lbl.

[0117] The generator 38 is a generator of a conditional GAN ​​model, and has the same function and configuration as the generator 22 of the learning processing device 200. The generator 38 stores a trained model (a model of a generator that has completed the learning process by the learning processing device 200 (optimal parameters are θ ｏｐｔ （Ｇ）The generator 38 is equipped with a map image data (condition image data) Do_map output from the map image generation unit 36. The generator 38 inputs the map image data (condition image data) Do_map into the trained model of the generator 38, executes processing using the trained model, and outputs the data acquired by the processing to the generated data acquisition processing unit 39 as data D_gene_out.

[0118] The generated data acquisition processing unit 39 receives the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37. The generated data acquisition processing unit 39 performs a process (generated data acquisition process) to combine the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37 into a set of data (paired data (labeled data)), and outputs the data acquired by this process to the second data storage unit DB2 as data Dout (= {D_gene_out, Lbl}).

[0119] (1.1.5: Second Data Storage Unit DB2) The second data storage unit DB2 is a functional unit that can read data from a predetermined area (memory area) of the second data storage unit DB2 in accordance with an external data read command and / or write data to a predetermined area (memory area) of the second data storage unit DB2 in accordance with an external data write command. The second data storage unit DB2 is realized by, for example, a data storage device, a database, or a memory device.

[0120] In accordance with a data write command from the medical image data generation device 300, the second data storage unit DB2 writes the data Dout output from the generated data acquisition processing unit 39 of the medical image data generation device 300 into a specified area (memory area) of the second data storage unit DB2.

[0121] <1.2: Operation of the Medical Diagnosis Training Data Generation System> The operation of the medical diagnosis training data generation system 1000 configured as above will be described with reference to the drawings.

[0122] In the following, the operation of the medical diagnosis training data generation system 1000 will be explained separately for (1) training data generation processing, (2) training processing, and (3) medical image data generation processing.

[0123] 5 and 6 are diagrams for explaining the processing (standardization processing) of the standardization unit 12. FIG.

[0124] FIG. 7 is a diagram for explaining the processing (feature extraction processing) of the feature extraction image generating unit 13. In FIG.

[0125] FIG. 8 is a diagram for explaining the processing (first editing processing) of the first editing processing unit 14. In FIG.

[0126] FIG. 9 is a diagram for explaining the processing (second editing processing) of the second editing processing unit 15. In FIG.

[0127] (1.2.1: Learning Data Generation Processing) First, the learning data generation processing will be described.

[0128] Actual medical image data (real medical image data) is input to the real image input unit 11 (for example, a user inputs real medical image data from outside to the real image input unit 11). The real image input unit 11 outputs data including the input medical image data to the standardization unit 12 as data Din_real.

[0129] The standardization unit 12 inputs the data Din_real output from the real image input unit 11, acquires the medical image data (data of the real medical image) included in the data Din_real, and performs a standardization process on the medical image data (data of the real medical image). For example, the standardization unit 12 performs standardization processes on the medical image data (data of the real medical image) by performing (1) a process of cutting out (extracting) a predetermined region (image region) of the image (e.g., a process of cutting out (extracting) the left and right regions (or the top and bottom regions) of the image), (2) a process of padding a predetermined region of the image (padding process), and (3) a process of resizing the image, and thereby acquires a standardized image (e.g., an image of a predetermined size) through the standardization process.

[0130] For example, when the image shown in Figure 5 is input to the standardization unit 12 as medical image data (data of an actual medical image), the standardization unit 12 performs (1) a process of cutting out the left and right regions of the image, (2) a padding process (a process of padding the left and right image regions with zeros after the cut-out process), and (3) a resizing process, thereby obtaining an image (the image shown on the right side of Figure 5) in which the left and right image regions of the input medical image data (data of an actual medical image) are cut out as a standard image (for example, an image with an image size of x1 x y1).

[0131] Furthermore, when the image shown in Figure 6 is input to the standardization unit 12 as medical image data (data of an actual medical image), the standardization unit 12 can obtain an image (the image shown on the right side of Figure 6) cut out from a specified image area of ​​the input medical image data (data of an actual medical image) as a standard image (for example, an image with an image size of x1 x y1) by performing (1) padding processing (processing to pad the left and right image areas with zeros after the cutout processing) and (2) resizing processing.

[0132] The standardization unit 12 outputs data including the data (image data) after the standardization process as data D1_real to the feature extraction image generation unit 13, the second editing processing unit 15, and the learning data generation unit 17.

[0133] The feature extraction image generation unit 13 inputs the data D1_real output from the standardization unit 12, performs a feature extraction image generation process on the image data after the standardization process included in the data D1_real, and generates (obtains) a feature extraction image through this process.

[0134] For example, when the standardized data (standardized image) D1_real shown in Fig. 7 is input to the feature extraction image generation unit 13, the feature extraction image generation unit 13 executes feature extraction image generation processing (e.g., edge extraction processing) on ​​the input standard image to obtain an image from which features have been extracted (e.g., an edge extraction image) (e.g., the image shown on the right side of Fig. 7). Note that in Fig. 7, the feature extraction image (e.g., the edge extraction image) is displayed in a negative / positive inverted state.

[0135] Then, the feature extraction image generation unit 13 outputs data including the feature extraction image generated (acquired) by the feature extraction image generation process to the first edit processing unit 14 and the learning data generation unit 17 as data D1_map_auto.

[0136] The first editing processing unit 14 receives data D1_map_auto output from the feature extraction image generation unit 13 and data D11_stylus (data D11_stylus is, for example, data input from an external source) that indicates the editing content. Based on the data D11_stylus that indicates the editing content, the first editing processing unit 14 performs editing processing on the feature extraction image included in the data D1_map_auto and obtains image data after the editing processing.

[0137] For example, when the feature extraction image data (e.g., edge extraction image) D1_map_auto shown in FIG. 8 is input to the first editing processing unit 14, the first editing processing unit 14 displays the feature extraction image data (e.g., edge extraction image) D1_map_auto on, for example, a touch panel display device (not shown), and performs editing processing on the feature extraction image data (e.g., edge extraction image) D1_map_auto by, for example, having the user draw or erase in a predetermined image area on the displayed image with a touch pen. The data of this editing processing is then input to the first editing processing unit 14 as data D11_stylus. The first editing processing unit 14 performs editing processing on the feature extraction image data (e.g., edge extraction image) D1_map_auto based on the data D11_stylus that indicates the editing content, and obtains data of the image after the editing processing (e.g., the image shown on the right side of FIG. 8 ). In FIG. 8, the feature extraction image (for example, edge extraction image) and post-edit image data are displayed in a negative / positive inverted state.

[0138] Then, the first edit processing unit 14 outputs data including the edited image data to the learning data generation unit 17 as data D11_map_edited.

[0139] The second editing processing unit 15 receives the standardized image data D1_real output from the standardization unit 12 and data D12_stylus (data D12_stylus is, for example, data input from an external source) that indicates the editing content. Based on the data D12_stylus that indicates the editing content, the second editing processing unit 15 performs editing processing on the standardized image included in the standardized image data D1_real and obtains the image data after the editing processing.

[0140] For example, when the standardized image data D1_real shown in FIG. 9 is input to the second editing processing unit 15, the second editing processing unit 15 displays the standardized image data D1_real on, for example, a touch panel display device (not shown), and performs an editing process on the standardized image data D1_real (for example, a process of obtaining image data created by drawing by tracing the edges of the standardized image) by having the user use a touch pen to draw (for example, by tracing the edges of the standardized image) or erase in a predetermined image area on the displayed image. The data from this editing process is then input to the second editing processing unit 15 as data D12_stylus. The second editing processing unit 15 performs an editing process on the standardized image data D1_real based on the data D12_stylus indicating the editing content, and obtains data on the image after the editing process (for example, the image shown on the right side of FIG. 9). Note that the image on the right side of FIG. 9 is displayed in a negative / positive inverted state.

[0141] Then, the second edit processing unit 15 outputs data including the edited image data to the learning data generation unit 17 as data D12_map_edited.

[0142] The manually generated image input unit 16 is a functional unit (e.g., a data input interface) for inputting manually generated image data, and outputs data including the input image data as data D1_map_manual to the training data generation unit 17. For example, the manually generated image input unit 16 has a touch panel display device (not shown), and on the touch panel display device, a user draws and creates an image similar to the feature extraction image data (e.g., an edge extraction image) D1_map_auto using, for example, a touch pen. Then, the manually generated image input unit 16 outputs data including the input image data (an image manually created by the user (an image similar to the feature extraction image data (e.g., an edge extraction image) D1_map_auto)) to the training data generation unit 17 as data D1_map_manual.

[0143] The learning data generation unit 17 inputs the data D1_real output from the standardization unit 12, the data D1_map_auto output from the feature extraction image generation unit 13, the data D11_map_edited output from the first edit processing unit 14, the data D12_map_edited output from the second edit processing unit 15, and the data D1_map_manual output from the manually generated image input unit 16. The learning data generation unit 17 also inputs a mode signal Mode from a control unit (not shown).

[0144] In accordance with the mode signal Mode, the learning data generation unit 17 pairs (1) the data D1_real output from the standardization unit 12 with one of (2A) the data D1_map_auto output from the feature extraction image generation unit 13, (2B) the data D11_map_edited output from the first editing processing unit 14, (2C) the data D12_map_edited output from the second editing processing unit 15, and (2D) the data D1_map_manual output from the manually generated image input unit 16, thereby obtaining learning data (learning data in which standardized real image data and map image data (condition image data) are paired).

[0145] For example, the learning data generation unit 17 performs the following processing in accordance with the signal value of the mode signal Mode: (1) When the signal value of the mode signal Mode is “1,” the learning data generation unit 17 acquires, as learning data Dtrain_pair, data (={D1_real, D1_map_auto}) that is a pair of data D1_real output from the standardization unit 12 and data D1_map_auto output from the feature extraction image generation unit 13. (2) When the signal value of the mode signal Mode is "2", the learning data generation unit 17 acquires, as the learning data Dtrain_pair, data (={D1_real, D11_map_edited}) obtained by pairing the data D1_real output from the standardization unit 12 with the data D11_map_edited output from the first editing processing unit 14. (3) When the signal value of the mode signal Mode is "3", the learning data generation unit 17 acquires, as the learning data Dtrain_pair, data (={D1_real, D12_map_edited}) obtained by pairing the data D1_real output from the standardization unit 12 with the data D12_map_edited output from the second editing processing unit 15. (4) When the signal value of the mode signal Mode is “4”, the learning data generation unit 17 acquires data (= {D1_real, D1_map_manual}) that is a pair of data D1_real output from the standardization unit 12 and data D1_map_manual output from the manually generated image input unit 16 as learning data Dtrain_pair.

[0146] Then, the learning data generating unit 17 outputs the acquired learning data Dtrain_pair to the first data storage unit DB1.

[0147] Then, in accordance with a data write command from the learning data generation device 100, the first data storage unit DB1 writes the data Dtrain_pair output from the learning data generation unit 17 of the learning data generation device 100 into a specified area (memory area) of the first data storage unit DB1.

[0148] (1.2.2: Learning Process) Next, the learning process will be described.

[0149] The data input unit 21 of the learning processing device 200 outputs a data read command to the first data storage unit DB1, reads out learning data (learning data in which standardized actual image data and map image data (condition image data) are paired) stored in a predetermined area (memory area) of the first data storage unit DB1, and inputs the read learning data as data Din_pair (learning data Din_pair). The data input unit 21 extracts map image data (condition image data) from the learning data Din_pair and outputs the extracted map image data (condition image data) as data Din_condition to the generator 22, the first combination unit 23A, and the second combination unit 23B.

[0150] Furthermore, the data input unit 21 extracts standardized real image data from the learning data Din_pair, and outputs the extracted standardized real image data to the second combining unit 23B as data Din_real.

[0151] A generator 22 (the generator parameters are θ （Ｇ） The data input unit 23A receives map image data (condition image data) Din_condition output from the data input unit 21. The generator 22 inputs the map image data (condition image data) Din_condition into the trainable model of the generator 22, executes processing using the trainable model, and outputs the data acquired by the processing to the first combination unit 23A as data Do_G.

[0152] The first combination unit 23A inputs the data Do_G output from the generator 22 and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Do_G and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sG.

[0153] The second combining unit 23B inputs the data Din_real and data Din_condition output from the data input unit 21, performs a process of concatenating the data Din_real and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sx.

[0154] The selector SEL1 selects either the data Do_sG output from the first combining unit 23A or the data Ds_sx output from the second combining unit 23B in accordance with a selection signal sel1 from a control unit (not shown) that controls the functional units of the learning processing device 200. When the signal value of the selection signal sel is "0," the selector SEL1 selects the data Do_G output from the generator 22 (data input to terminal 0) and outputs the data Do_G as data Do0 to the downsampling unit 24 and the second classifier 25B. On the other hand, when the signal value of the selection signal sel is "1," the selector SEL1 selects the data Do_sx output from the second combining unit 23B (data input to terminal 1) and outputs the data Do_sx as data Do0 to the downsampling unit 24 and the second classifier 25B.

[0155] The downsampling unit 24 inputs data Do0 output from the selector SEL1, performs a downsampling process on the data Do0, obtains the downsampled data, and outputs the obtained data as data Do1 to the first identifier 25A.

[0156] A first discriminator 25A (the discriminator parameters are θ （Ｄ１）The first classifier 25A inputs the data Do1 output from the downsampling unit 24 to its trainable model, executes processing by the trainable model (processing for determining whether the input data Do1 is genuine or fake), and outputs data including the determination result data obtained by the processing and the output data (feature data) of the T layers (layers for extracting features) (T: natural number) as data Do_D1 to the loss evaluation unit 26. The trainable model of the first classifier 25A outputs "1" when it determines that the input data Do1 is genuine, and outputs "0" when it determines that the input data D2 is fake.

[0157] A second discriminator 25B (with discriminator parameters θ （Ｄ２） The second classifier 25B inputs the data Do0 output from the selector SEL1 to its trainable model, executes processing by the trainable model (processing for determining whether the input data Do0 is genuine or fake), and outputs data including the determination result data obtained by the processing and the output data (feature data) of the T layers (layers for extracting features) (T: natural number) as data Do_D2 to the loss evaluation unit 26. Note that the trainable model of the second classifier 25B outputs "1" if it determines that the input data Do1 is genuine, and outputs "0" if it determines that the input data D2 is fake.

[0158] In addition, the teacher label (corresponding to the teacher data D_correct) for the judgment data as to whether it is genuine or fake, contained in the output data Dout_D1 and Dout_D2 of the trainable models of the first classifier 25A and the second classifier 25B, is “1” indicating that it is genuine data when data including real data (including data after downsampling) is input to the first classifier 25A and the second classifier 25B, and is “0” indicating that it is fake data when data including fake data (output data of the generator 22) (including data after downsampling) is input to the first classifier 25A and the second classifier 25B.

[0159] The loss evaluation unit 26 obtains the determination result data Do_D1 output from the first classifier 25A, the determination result data Do_D2 output from the second classifier 25B, an error (loss obtained by a loss function) based on the determination result data Do_D1 and the teacher data D_correct, and an error (loss obtained by a loss function) based on the determination result data Do_D2 and the teacher data D_correct. Then, the loss evaluation unit 26 performs parameter update processing (learning processing) of the trainable models of the first classifier 25A, the second classifier 25B, and the generator 22 based on the obtained errors (losses) (for example, learning processing using an error backpropagation method).

[0160] Specifically, the learning processing device 200 performs the learning process by performing processing corresponding to the following equations. λ: coefficient D ｋ : k-th classifier (k: integer, k={1, 2}) Note that the loss function L for the generator G (generator 22) and the classifier D (first classifier 25A and second classifier 25B) ＧＡＮ (G.D. ｋ ) is as follows: s: condition image data (corresponding to data Din_condition) (probability distribution) x: actual image data (real image data) (probability distribution) G(s): data (probability distribution) generated by the generator 22 when the condition image data s is input to the generator 22 D(s, x): data (probability distribution) generated by the classifier when the condition image data s and actual image data (real image data) x are input to the classifier E[·]: expected value ||·|| １ : L1 norm That is, the learning processing device 200 performs processing corresponding to the above formula to obtain the loss function L for the generator G (generator 22) and the classifier D (first classifier 25A and second classifier 25B). ＧＡＮ (G.D. ｋ ) to obtain the

[0161] Furthermore, the feature matching loss function LFM(G, D) for the generator G (generator 22) and the k-th classifier (the first classifier 25A and the second classifier 25B) is ｋ ) is as follows: E[·]: Expected value ||·|| １ : L1 norm G(s): Data (probability distribution) generated by the generator 22 when the condition image data s is input to the generator 22 T: Number of layers (feature extraction layers) of the first classifier 25A (= Number of layers (feature extraction layers) of the second classifier 25B) D ｋ （ｉ） (s, x): Data (probability distribution) output from the i-th layer of the k-th classifier when condition image data s and real image data (real image data) x are input to the k-th classifier (k: integer, k = {1, 2}). D ｋ （ｉ） (s, G(s)): Data (probability distribution) output from the i-th layer of the k-th classifier when the condition image data s and fake image data G(s) (output data G(s) from the generator 22) are input to the k-th classifier (k: integer, k={1, 2}). In other words, the learning processing device 200 performs processing equivalent to the above formula to calculate the feature matching loss function L for the generator G (generator 22) and the k-th classifier (first classifier 25A and second classifier 25B). ＦＭ (G.D. ｋ ) to obtain the

[0162] Then, (1) the loss function L for the generator G (the generator 22) and the classifier D (the first classifier 25A and the second classifier 25B) obtained as described above is calculated. ＧＡＮ (G.D. ｋ ), and (2) a feature matching loss function L for the generator G (generator 22) and the k-th classifier (the first classifier 25A and the second classifier 25B). ＦＭ (G.D. ｋ ), the learning processing device 200 performs processing equivalent to (Equation 1), thereby generating the parameter θ （Ｇ） update process of the parameter θ of the first classifier 25A （Ｄ１） and the parameter θ of the second classifier 25B （Ｄ２） The parameter θ of the generator 22 is updated. （Ｇ） The optimal parameter θ ｏｐｔ （Ｇ） , the parameter θ of the first classifier 25A （Ｄ１） The optimal parameter θ ｏｐｔ （Ｄ１） , and the parameter θ of the second classifier 25B （Ｄ２） The optimal parameter θ ｏｐｔ （Ｄ２） In the learning processing device 200, the parameter θ （Ｇ） The optimal parameter θ ｏｐｔ （Ｇ） , the parameter θ of the first classifier 25A （Ｄ１） The optimal parameter θ ｏｐｔ （Ｄ１） , and the parameter θ of the second classifier 25B （Ｄ２） The optimal parameter θ ｏｐｔ （Ｄ２） Once this is acquired, the learning process is terminated.

[0163] (1.2.3: Medical Image Data Generation Processing) Next, the medical image data generation processing will be described.

[0164] The generator 38 of the medical image data generating device 300 is provided with the optimal parameters θ of the generator acquired by the learning processing of the learning processing device 200. ｏｐｔ （Ｇ） It is equipped with a model (pre-trained model) with the following settings.

[0165] Actual medical image data (real medical image data) is input to the second real image input unit 30 (for example, a user inputs real medical image data from outside into the second real image input unit 30). The second real image input unit 30 outputs data including the input medical image data to the second standardization unit 31 as data Din2_real.

[0166] The second standardization unit 31 inputs the data Din2_real output from the second real image input unit 30, acquires the medical image data (data of the real medical image) included in the data Din2_real, and performs a standardization process on the medical image data (data of the real medical image). For example, the second standardization unit 31 performs standardization processes on the medical image data (data of the real medical image) by performing (1) a process of cutting out (extracting) a predetermined region (image region) of the image (e.g., a process of cutting out (extracting) the left and right regions (or the top and bottom regions) of the image), (2) a process of padding the predetermined region of the image (padding process), and (3) a process of resizing the image, and thereby acquires a standardized image (e.g., an image of a predetermined size) (performing the same process as the standardization unit 12).

[0167] Then, the second standardization unit 31 outputs data including the standardized data (image data) as data D2_real to the second feature extraction image generation unit 32 and the fourth edit processing unit 34.

[0168] The second feature extraction image generation unit 32 inputs the data D2_real output from the second standardization unit 31, performs a feature extraction image generation process on the image data after the standardization process included in the data D2_real, and generates (obtains) a feature extraction image through this process (performing the same process as the feature extraction image generation unit 13).

[0169] Then, the second feature extraction image generating unit 32 outputs data including the feature extraction image generated (acquired) by the feature extraction image generating process to the third edit processing unit 33 and the map image generating unit 36 ​​as data D2_map_auto.

[0170] The third edit processing unit 33 receives data D2_map_auto output from the second feature extraction image generation unit 32 and data D21_stylus (data D21_stylus is, for example, data input from an external source) that indicates the editing content. Based on the data D21_stylus that indicates the editing content, the third edit processing unit 33 performs editing processing on the feature extraction image included in the data D2_map_auto and obtains image data after the editing processing (performing processing similar to that of the first edit processing unit 14).

[0171] Then, the third edit processing unit 33 outputs data including the image data after the edit processing to the map image generating unit 36 ​​as data D21_map_edited.

[0172] The fourth editing processing unit 34 receives the standardized image data D2_real output from the second standardization unit 31 and data D22_stylus indicating the editing content (the data D22_stylus is, for example, data input from an external device). The fourth editing processing unit 34 performs editing processing on the standardized image included in the standardized image data D2_real based on the data D22_stylus indicating the editing content, and obtains the image data after the editing processing (performing the same processing as the second editing processing unit 15).

[0173] Then, the fourth edit processing unit 34 outputs data including the edited image data to the map image generating unit 36 ​​as data D22_map_edited.

[0174] The second manually generated image input unit 35 is a functional unit (e.g., a data input interface) for inputting manually generated image data, and outputs data including the input image data to the map image generation unit 36 ​​as data D2_map_manual. For example, the second manually generated image input unit 35 has a touch panel display device (not shown), and on the touch panel display device, a user draws and creates an image similar to the feature extraction image data (e.g., an edge extraction image) D2_map_auto using, for example, a touch pen. Then, the second manually generated image input unit 35 outputs data including the input image data (an image manually created by the user (an image similar to the feature extraction image data (e.g., an edge extraction image) D2_map_auto)) to the map image generation unit 36 ​​as data D2_map_manual.

[0175] Map image generation unit 36 ​​receives data D2_map_auto output from second feature amount extraction image generation unit 32, data D21_map_edited output from third edit processing unit 33, data D22_map_edited output from fourth edit processing unit 34, and data D2_map_manual output from second manually generated image input unit 35. Map image generation unit 36 ​​also receives a mode signal Mode from a control unit (not shown).

[0176] In accordance with the mode signal Mode, the map image generation unit 36 ​​acquires one of the following data as map image data Do_map to be output to the generator 38: (1) data D2_map_auto output from the second feature extraction image generation unit 32, (2) data D21_map_edited output from the third editing processing unit 33, (3) data D22_map_edited output from the fourth editing processing unit 34, and (4) data D2_map_manual output from the second manually generated image input unit 35.

[0177] For example, the learning data generation unit 17 performs the following processing according to the signal value of the mode signal Mode: (1) When the signal value of the mode signal Mode is "1," the map image generation unit 36 ​​acquires data D2_map_auto output from the second feature amount extraction image generation unit 32 as map image data Do_map. (2) When the signal value of the mode signal Mode is "2," the map image generation unit 36 ​​acquires data D21_map_edited output from the third edit processing unit 33 as map image data Do_map. (3) When the signal value of the mode signal Mode is "3," the map image generation unit 36 ​​acquires data D22_map_edited output from the fourth edit processing unit 34 as map image data Do_map. (4) When the signal value of the mode signal Mode is "4", the map image generating unit 36 ​​acquires the data D2_map_manual output from the second manually generated image input unit 35 as the map image data Do_map.

[0178] Then, map image generating unit 36 ​​outputs the map image data Do_map obtained as described above to generator 38 .

[0179] The generator 38 is provided with a trained model (a model of a generator that has completed training processing by the training processing device 200 (optimal parameters are θ ｏｐｔ （Ｇ） The generator 38 is equipped with a map image data (condition image data) Do_map output from the map image generation unit 36. The generator 38 inputs the map image data (condition image data) Do_map into the trained model of the generator 38, executes processing using the trained model, and outputs the data acquired by the processing to the generated data acquisition processing unit 39 as data D_gene_out.

[0180] Data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 ​​is input to the label input unit 37. Then, the label input unit 37 outputs the input data (label data) as data Lbl to the generated data acquisition processing unit 39. Note that the user knows in advance what kind of data the actual image data input to the second actual image input unit 30 or the data input to the second manually generated image input unit 35 is (the user knows in advance that the actual image data input to the second actual image input unit 30 or the data input to the second manually generated image input unit 35 is, for example, medical image data of a subject suffering from a specific disease or medical image data of a healthy state), and therefore can accurately input data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 ​​to the label input unit 37.

[0181] The generated data acquisition processing unit 39 receives the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37. The generated data acquisition processing unit 39 performs a process (generated data acquisition process) to combine the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37 into a set of data (paired data (labeled data)), and outputs the data acquired by this process to the second data storage unit DB2 as data Dout (= {D_gene_out, Lbl}).

[0182] By repeatedly executing the above process in the medical image data generation device 300, a large amount of highly accurate labeled medical image data can be generated.

[0183] Summary As described above, in the medical diagnosis training data generation system 1000, the training data generation device 100 can acquire a variety of training image data, namely: (1) training data (training data in automatic generation mode) that combines standardized actual image data with map image (condition image) data (image data from which features have been extracted) that is automatically generated from the actual image data; (2) training data (training data in semi-automatic mode (map image editing mode)) that combines standardized actual image data with map image (condition image) data obtained by editing a map image that is automatically generated from the actual image data; (3) training data (training data in semi-automatic mode (actual image editing mode)) that combines standardized actual image data with map image (condition image) data obtained by performing an editing process based on the actual image data; and (4) training data (training data in manual mode) that combines standardized actual image data with map image (condition image) data that is manually generated. In the medical diagnostic training data generation system 1000, a trainable model (e.g., a conditional GAN ​​model) can be trained in the learning processing device 200 using a variety of training image data acquired by the training data generation device 100. In other words, in the training process of the medical diagnostic training data generation system 1000, a trainable model (e.g., a conditional GAN ​​model) can be trained in the learning processing device 200 using actual image data, which can only be acquired in small amounts, and map image data acquired based on the actual image data. Therefore, even if only a small amount of actual image data can be acquired, a large amount of training data for training the trainable model can be acquired. As a result, the learning processing device 200 can train a trainable model (e.g., a conditional GAN ​​model) with high accuracy.

[0184] In the medical diagnostic training data generation system 1000, the medical image data generation device 300 can acquire a variety of input image data (input data for inference processing) based on a small number of actual image data, namely: (1) map image (condition image) data (image data from which features have been extracted) automatically generated from standardized actual image data (input data for inference in automatic generation mode), (2) map image (condition image) data acquired by editing a map image automatically generated from standardized actual image data (input data for inference in semi-automatic mode (map image editing mode)), (3) map image (condition image) data acquired by performing an editing process based on the standardized actual image data (input data for inference in semi-automatic mode (actual image editing mode)), and (4) map image (condition image) data generated manually (input data for inference in manual mode). In the medical diagnostic training data generation system 1000, the various input data for inference (image data) acquired as described above can be input into the generator 38 equipped with a trained model, thereby acquiring highly accurate generated image data D_gene_out. Furthermore, in the medical diagnostic training data generation system 1000, the user knows in advance what kind of data the actual image data input to the second actual image input unit 30 of the medical image data generation device 300 or the data input to the second manually-generated image input unit 35 is (the user knows in advance that the actual image data input to the second actual image input unit 30 or the data input to the second manually-generated image input unit 35 is, for example, medical image data of a subject suffering from a specific disease or medical image data of a healthy state), so data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 ​​can be accurately input to the label input unit 37. The generated data acquisition processing unit 39 of the medical image data generation device 300 can then acquire data (paired data (labeled data)) Dout, which is a pair of data D_gene_out output from the generator 38 and data Lbl output from the label input unit 37. This makes it possible to easily acquire large amounts of accurately annotated data (labeled data).

[0185] As described above, the medical diagnostic training data generation system 1000 can collect a large amount of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation.

[0186] [Other Embodiments] In the above embodiment, a case has been described in which a conditional GAN ​​model is employed in the learning processing device 200 of the medical diagnostic training data generation system 1000. However, this is not limited to this, and other generation models that can accept condition input (e.g., a diffusion model that can accept condition input) may also be employed. In other words, the learning processing device 200 may perform a learning process using other generation models that can accept condition input (e.g., a diffusion model that can accept condition input), and the trained model acquired by this learning process may be loaded into the medical image data generation device 300, which may then perform inference processing (medical image data generation processing).

[0187] In the medical diagnostic training data generation system 1000 described in the above embodiment, each block (each functional unit) may be individually implemented as a single chip using a semiconductor device such as an LSI, or may be integrated into a single chip to include some or all of the blocks. Furthermore, each block (each functional unit) of the medical diagnostic training data generation system 1000, training data generation device 100, training processing device 200, and medical image data generation device 300 described in the above embodiment may be realized by multiple semiconductor devices such as LSIs.

[0188] Although the term "LSI" is used here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

[0189] Furthermore, the method of integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI.

[0190] Furthermore, some or all of the processing of each functional block in each of the above embodiments may be realized by a program. And, some or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Furthermore, the programs for performing each processing are stored in a storage device such as a hard disk or ROM, and are read out and executed in the ROM or RAM.

[0191] Furthermore, each process in the above-described embodiment may be realized by hardware, or by software (including cases where it is realized together with an OS (operating system), middleware, or a predetermined library). Furthermore, it may be realized by a combination of software and hardware.

[0192] When each functional unit of the above embodiment is realized by software, for example, each functional unit may be realized by software processing using the hardware configuration shown in FIG. 10 (for example, a hardware configuration in which a CPU (which may be a GPU), a ROM, a RAM, an input unit, an output unit, a communication unit, a memory unit (for example, a memory unit realized by an HDD, an SSD, etc.), an external media drive, etc. are connected via a bus).

[0193] Furthermore, when each functional unit of the above embodiment is realized by software, the software may be realized using a single computer having the hardware configuration shown in Figure 10, or may be realized by distributed processing using multiple computers.

[0194] Furthermore, the execution order of the processing method in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed within the scope of the gist of the invention. Furthermore, in the processing method in the above embodiment, some steps may be executed in parallel with other steps within the scope of the gist of the invention.

[0195] The scope of the present invention includes a computer program for causing a computer to execute the above-described method, and a computer-readable recording medium on which the program is recorded. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories.

[0196] The computer program is not limited to one recorded on the recording medium, but may be one transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, or the like.

[0197] The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention.

[0198] 1000 Medical diagnosis training data generation system 100 Training data generation device 200 Training processing device 300 Medical image data generation device 11 Actual image input unit 12 Standardization unit 13 Feature extracted image generation unit 14 First editing processing unit 15 Second editing processing unit 16 Manually generated image input unit 17 Training data generation unit 21 Data input unit 22 Generator 25A First classifier 25B First classifier 26 Loss evaluation unit 30 Second actual image input unit 31 Second standardization unit 32 Second feature extracted image generation unit 33 Third editing processing unit 34 Fourth editing processing unit 35 Second manually generated image input unit 36 ​​Map image generation unit 37 Label input unit 38 Generator (inference generator) 39 Generated data acquisition processing unit

Claims

1. An image input unit for inputting image data; a standardization unit for performing standardization processing on image data input to the image input unit to obtain standardized image data; a feature extraction image generation unit for performing feature extraction processing on the standardized image data to obtain feature extraction image data; a first edit processing unit capable of inputting first edit data, which is data for editing the feature extraction image data, and performing edit processing on the feature extraction image data based on the first edit data, and obtaining the image data after the edit processing as the first edited image data; a second edit processing unit capable of inputting second edit data, which is data for editing the standardized image data, and performing edit processing on the standardized image data based on the second edit data, and obtaining the image data after the edit processing as the second edited image data; a manually generated image input unit for inputting manually generated image data, which is data for a map image that is manually generated; and a learning data generation unit for obtaining learning data that is a combination of the standardized image data and any one of image data of (1) the feature extraction image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data. A learning data generation device comprising:

2. A learning processing device for performing learning processing using learning data generated by the learning data generation device described in claim 1, comprising: a data input unit that inputs the learning data generated by the learning data generation device described in claim 1 and extracts condition image data and actual image data from the learning data; a learning processing unit that includes a trainable model, inputs the condition image data and the actual image data, and performs learning processing of the trainable model; and a loss evaluation unit that evaluates the error between data output from the learning processing unit and teacher data.

3. The learning processing device according to claim 2, wherein the learning processing unit comprises: a generator having a trainable model that inputs the condition image and generates image data based on the input condition image; and a classifier that inputs the condition image and either the image data generated by the generator or the actual image data, and outputs a determination process result as to whether the input data is real data or fake data.

4. An image input unit for inputting image data; a standardization unit for performing standardization processing on image data input to the image input unit to obtain standardized image data; a feature extraction image generation unit for performing feature extraction processing on the standardized image data to obtain feature extraction image data; a first edit processing unit capable of inputting first edit data, which is data for editing the feature extraction image data, and performing edit processing on the feature extraction image data based on the first edited image, and obtaining the image data after the edit processing as the first edited image data; a second edit processing unit capable of inputting second edit data, which is data for editing the standardized image data, and performing edit processing on the standardized image data based on the second edited image, and obtaining the image data after the edit processing as the second edited image data; a manually generated image input unit for inputting manually generated image data, which is data for a map image that is manually generated; and a map image generation unit for obtaining image data of any one of (1) the feature extraction image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data as map image data. an image data generation device equipped with a trained model of the learnable model obtained by a learning process by the learning processing device of claim 2, or a trained model of the learnable model of the generator of the learning processing device of claim 3 obtained by a learning process by the learning processing device of claim 3, and comprising: an inference generator that generates inference processing result image data from the map image data; a label input unit that inputs a label for the map image data; and a generated data acquisition processing unit that acquires labeled image data that pairs the label input to the label input unit with the inference processing result image data generated by the inference generator.

5. An image input step for inputting image data; a standardization step for performing standardization processing on the image data input in the image input step to obtain standardized image data; a feature extraction image generation step for performing feature extraction processing on the standardized image data to obtain feature extraction image data; a first editing processing step for inputting first editing data, which is data for editing the feature extraction image data, and performing editing processing on the feature extraction image data based on the first editing data, and obtaining the image data after the editing processing as the first edited image data; a second editing processing step for inputting second editing data, which is data for editing the standardized image data, and performing editing processing on the standardized image data based on the second editing data, and obtaining the image data after the editing processing as the second edited image data; a manually generated image input step for inputting manually generated image data, which is data for a map image that is manually generated; and a learning data generation step for obtaining learning data that is a combination of the standardized image data and any one of (1) the feature extraction image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data. A learning data generation method comprising:

6. A learning processing method for performing learning processing using learning data generated by the learning data generation method of claim 5, comprising: a data input step of inputting the learning data generated by the learning data generation method of claim 5 and extracting condition image data and actual image data from the learning data; a learning processing step of inputting the condition image data and the actual image data and performing learning processing of a trainable model; and a loss evaluation step of evaluating the error between the data acquired by the learning processing step and teacher data.

7. The learning processing method according to claim 6, wherein the learning processing step comprises: a generation step of inputting the condition image and generating image data using a learnable model based on the input condition image; and an identification step of inputting the condition image and either the image data generated by the generation step or the actual image data and outputting a determination processing result as to whether the input data is real data or fake data.

8. An image input step for inputting image data; a standardization step for performing standardization processing on the image data input in the image input step to obtain standardized image data; a feature extraction image generation step for performing feature extraction processing on the standardized image data to obtain feature extraction image data; a first editing processing step for inputting first editing data, which is data for editing the feature extraction image data, and performing editing processing on the feature extraction image data based on the first editing image, and obtaining the image data after the editing processing as the first edited image data; a second editing processing step for inputting second editing data, which is data for editing the standardized image data, and performing editing processing on the standardized image data based on the second editing image, and obtaining the image data after the editing processing as the second edited image data; a manually generated image input step for inputting manually generated image data, which is data for a map image that has been manually generated; and a map image generation step for obtaining image data of any one of (1) the feature extraction image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data as map image data. an inference generation step of generating inference processing result image data from the map image data using a trained model of the learnable model obtained by a learning process by the learning processing method of claim 6, or a trained model of the learnable model used in the generation step of the learning processing method of claim 7 obtained by a learning process by the learning processing method of claim 7; a label input unit that inputs a label for the map image data; and a generated data acquisition processing step of acquiring labeled image data that combines the label input to the label input unit with the inference processing result image data generated by the inference generation step.

9. A program for causing a computer to execute the learning data generation method according to claim 5.

10. A program for causing a computer to execute the learning processing method according to claim 6 or 7.

11. A program for causing a computer to execute the image data generating method according to claim 8.

Images