Learning data generating device, learning processing device, image data generating device, learning data generating method, learning processing method, image data generating method, and program

Abstract

We will realize a medical diagnostic learning data generation system that can collect large amounts of highly accurate learning data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation. [Solution] In a medical diagnosis training data generation system 1000, a variety of training image data can be acquired by a training data generation device 100. In the medical diagnosis training data generation system 1000, a trained model is acquired by performing a training process in a training processing device 200 using the variety of training image data acquired by the training data generation device 100. Then, the trained model is loaded into a medical image data generation device 300, labels are input, and inference processing is performed, thereby making it possible to generate a large amount of highly accurate labeled medical image data.

Description

[Technical field]

[0001] The present invention relates to a technique for acquiring large amounts of highly accurate learning data to be used for learning to generate a medical AI model. [Background technology]

[0002] In recent years, technology has been developed in which medical images (e.g., X-ray images) are input into a trained model obtained through machine learning, and a diagnostic result is output from the trained model (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-32071 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional techniques described above, machine learning needs to be performed using medical diagnostic images, but in order to obtain a high-performance trained model through this machine learning, a large amount of highly accurate training data (medical diagnostic images) is required.

[0005] However, it is difficult to collect a large amount of highly accurate learning data (medical diagnostic images (medical data)) for the following reasons.

[0006] Medical data is individualized: each patient has their own symptoms and constitution, and there is a lack of sufficient data to cover that diversity.

[0007] In addition, medical data is extremely sensitive and requires strict security measures when it is handled, making it difficult to collect large amounts of medical data.

[0008] In addition, medical data has the difficulty of being shared: the complexity of sharing medical data while protecting patient privacy is a barrier to data collection.

[0009] Furthermore, it is difficult to perform annotation of learning data (medical diagnostic images (medical data)) at low cost. In other words, annotation of medical data (medical diagnostic images) requires highly specialized knowledge and requires the judgment of an expert. As a result, annotation of learning data (medical diagnostic images (medical data)) takes time and costs money.

[0010] As such, it is difficult to collect a large amount of highly accurate training data (medical diagnostic images) required to obtain a high-performance trained model through machine learning, and to perform accurate annotation.

[0011] In view of the above problems, the present invention aims to provide a medical diagnostic training data generation system, a training data generation device, a training processing device, an image data generation device, a training data generation method, a training processing method, an image data generation method, and a program that can collect a large amount of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and that can perform accurate annotation. [Means for solving the problem]

[0012] In order to solve the above problems, a first invention is a learning data generation device including an image input unit, a standardization unit, a feature extraction image generation unit, a first editing processing unit, a second editing processing unit, a manually generated image input unit, and a learning data generation unit.

[0013] The image input unit inputs image data.

[0014] The standardization unit acquires standardized image data by performing a standardization process on the image data input to the image input unit.

[0015] The feature extraction image generating unit acquires feature extraction image data by performing feature extraction processing on the standardized image data.

[0016] The first editing processing unit can input first editing data, which is data for editing the feature extracted image data, and performs editing processing on the feature extracted image data based on the first editing data, and obtains the image data after the editing processing as the first edited image data.

[0017] The second editing processing unit can input second editing data, which is data for editing, to the standardized image data, perform editing processing on the standardized image data based on the second editing data, and obtain the image data after the editing processing as the second editing image data. The manually generated image input unit is a functional unit for inputting manually generated image data, which is data for a manually generated map image.

[0018] The learning data generating unit generates the standardized image data, (1) the feature extraction image data, (2) the first edited image data, and (3) the second edited image data. 2 (4) The edited image data and any one of the manually generated image data are paired to obtain learning data.

[0019] In this learning data generation device, by using the above configuration, various learning image data, i.e. (1) Learning data consisting of a combination of standardized image data (actual image data) and map image (condition image) data (image data from which features have been extracted) automatically generated from the actual image data; (2) Learning data consisting of a combination of standardized image data (actual image data) and map image (condition image) data obtained by editing a map image automatically generated from the actual image data; (3) Learning data that is a combination of standardized image data (actual image data) and map image (condition image) data obtained by performing editing processing based on the actual image data; (4) Learning data consisting of a combination of standardized image data (actual image data) and manually generated map image (condition image) data; can be acquired. Then, by using various learning image data acquired by this learning data generation device to train a trainable model (e.g., a conditional GAN ​​model), highly accurate learning processing can be performed. Then, by performing inference processing using the trained model acquired by the learning processing, a large amount of highly accurate generated image data (e.g., medical image data) can be acquired even when only a small amount of actual image data can be acquired.

[0020] Note that the "standardization process" refers to a process in which, for example, an input image is subjected to an extraction process of a specified image area, a padding process of the specified image area, and an image resizing process, thereby obtaining standardized image data.

[0021] A second invention is the first invention, and is a learning processing device including a data input unit, a learning processing unit, and a loss evaluation unit.

[0022] The data input unit inputs the learning data generated by the learning data generating device of the first invention, and extracts condition image data and actual image data from the learning data.

[0023] The learning processing unit includes a trainable model, receives the condition image data and the actual image data, and performs a learning process for the trainable model.

[0024] The loss evaluation unit evaluates the error between the data output from the learning processing unit and the teacher data.

[0025] As a result, in this learning processing device, the learning process for a trainable model can be performed using the learning data generated by the learning data generating device of the first invention. That is, even if only a small amount of actual image data can be acquired, the learning process for acquiring a trained model that can acquire a large amount of highly accurate generated image data (e.g., medical image data) can be efficiently performed.

[0026] A third invention is the second invention, in which the learning processing unit includes a generator and a classifier.

[0027] The generator is a condition image data Enter the condition image data The present invention provides a trainable model for generating image data based on the trained model.

[0028] The classifier is the condition image data Either image data generated by a generator or real image data is input, and the result of a determination process as to whether the input data is real data or fake data is output.

[0029] As a result, this learning processing device can employ, for example, a conditional GAN ​​model as a trainable model and perform learning processing.

[0030] In addition, the condition image data The input method of either the image data generated by the generator or the actual image data is, for example, data and the image data generated by the generator, or a condition image data and the actual image data may be input as concatenated data.

[0031] Furthermore, there may be a plurality of classifiers, and data with different scales (for example, downsampled data) may be input to the plurality of classifiers.

[0032] A fourth invention is an image data generation device including an image input unit, a standardization unit, a feature extraction image generation unit, a first edit processing unit, a second edit processing unit, a manually generated image input unit, a map image generation unit, an inference generator, a label input unit, and a generated data acquisition processing unit.

[0033] The image input unit inputs image data.

[0034] The standardization unit acquires standardized image data by performing a standardization process on the image data input to the image input unit.

[0035] The feature extraction image generating unit acquires feature extraction image data by performing feature extraction processing on the standardized image data.

[0036] The first edit processing unit is capable of inputting first edit data, which is data for editing the feature extraction image data, and applying the first edit data to the feature extraction image data. data and obtains the image data after the editing process as the first edited image data.

[0037] The second edit processing unit can input second edit data, which is data for editing the standardized image data, and performs the second edit processing on the standardized image data. data and obtains the image data after the editing process as second edited image data.

[0038] The manually generated image input unit inputs manually generated image data, which is data for a manually generated map image.

[0039] The map image generating unit receives (1) the feature extraction image data, (2) the first edited image data, and (3) the second edited image data. 2 Either one of the edited image data and (4) the manually generated image data is acquired as map image data.

[0040] The inference generator is equipped with a trained model of a learnable model obtained by a learning process by the learning processing device described in the second invention, or a trained model of a learnable model of the generator of the learning processing device described in the third invention obtained by a learning process by the learning processing device described in the third invention, and generates inference processing result image data from map image data.

[0041] The label input unit inputs a label for the map image data.

[0042] The generated data acquisition processing unit acquires labeled image data that is a combination of the label input to the label input unit and the inference processing result image data generated by the inference generator.

[0043] In this image data generating device, the above-mentioned configuration generates various input image data (input data for inference processing) based on a small number of actual image data, i.e., (1) Map image (condition image) data (image data with extracted features) automatically generated from standardized actual image data, (2) Map image (condition image) data obtained by editing map images automatically generated from standardized real image data; (3) Map image (condition image) data obtained by editing the standardized actual image data; (4) Manually generated map image (condition image) data, can be obtained. In this image data generating device, the various inference input data (image data) obtained as described above can be input to an inference generator equipped with a trainable model, thereby obtaining highly accurate generated image data. Furthermore, in the image data generating device, the user knows in advance what kind of data the image data (real image data) input to the second real image input unit 30 or the data input to the manually generated image input unit is (the user knows in advance that the real image data input to the image input unit or the data input to the manually generated image input unit is, for example, medical image data of a specific disease or medical image data of a health state), so that data (information) for identifying the label corresponding to the map image output from the map image generating unit can be accurately input to the label input unit. In addition, the generated data acquisition processing unit of this image data generating device can obtain data (pair data (labeled data)) in which the generated image data output from the inference generator and the label data output from the label input unit are paired, so that data that has been accurately annotated (labeled data) can be obtained easily and in large quantities.

[0044] Therefore, this image data generating device can collect large amounts of highly accurate learning data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation. A fifth invention is a learning data generation method including an image input step, a standardization step, a feature extraction image generation step, a first editing processing step, a second editing processing step, a manually generated image input step, and a learning data generation step.

[0045] The image input step inputs image data.

[0046] The standardization step performs a standardization process on the image data input in the image input step to obtain standardized image data.

[0047] The feature extraction image generating step performs feature extraction processing on the standardized image data to obtain feature extraction image data.

[0048] The first editing processing step can input first editing data, which is data for editing, to the feature extracted image data, perform editing processing on the feature extracted image data based on the first editing data, and obtain the image data after the editing processing as the first edited image data. The second editing processing step can input second editing data, which is data for editing, to the standardized image data, perform editing processing based on the second editing data on the standardized image data, and obtain the image data after the editing processing as the second editing image data.

[0049] The manually generated image input step inputs manually generated image data, which is data for a manually generated map image.

[0050] The learning data generating step includes: generating standardized image data; (1) feature extraction image data; (2) first edited image data; and (3) second edited image data. 2 (4) The edited image data and any one of the manually generated image data are paired to obtain learning data.

[0051] This makes it possible to realize a learning data generating method that has the same effects as the first aspect of the invention.

[0052] The sixth invention is a learning processing method for performing learning processing using learning data generated by the learning data generation method of the fifth invention, the learning processing method including a data input step, a learning processing step, and a loss evaluation step. The data input step inputs the learning data generated by the learning data generating method according to the fifth aspect of the invention, and extracts condition image data and actual image data from the learning data. The learning process step inputs the condition image data and the actual image data, and performs learning process for the trainable model. The loss evaluation step evaluates the error between the data acquired in the learning process step and the training data.

[0053] This makes it possible to realize a learning processing method that has the same effects as the second aspect of the invention.

[0054] A seventh invention is the sixth invention, in which the learning process step includes a generating step and an identifying step.

[0055] The generation step is to generate a condition image data Enter the condition image data Based on this, image data is generated using a trainable model.

[0056] The identification step is data Either the image data generated in the generation step or the actual image data is input, and the result of the determination process as to whether the input data is real data or fake data is output.

[0057] This makes it possible to realize a learning processing method that provides the same effects as the third aspect of the invention.

[0058] An eighth invention is an image data generation method including an image input step, a standardization step, a feature extraction image generation step, a first editing processing step, a second editing processing step, a manually generated image input step, a map image generation step, an inference generation step, a label input unit, and a generated data acquisition processing step.

[0059] The image input step inputs image data.

[0060] The standardization step performs a standardization process on the image data input in the image input step to obtain standardized image data.

[0061] The feature extraction image generating step performs feature extraction processing on the standardized image data to obtain feature extraction image data.

[0062] In the first editing process step, first editing data, which is data for editing the feature extraction image data, can be input, and the first editing data can be applied to the feature extraction image data. data and obtains the image data after the editing process as the first edited image data. In the second editing process step, second editing data, which is data for editing the standardized image data, can be input, and the second editing data can be applied to the standardized image data. data and obtains the image data after the editing process as second edited image data.

[0063] The manually generated image input step inputs manually generated image data, which is data for a manually generated map image.

[0064] The map image generating step includes: (1) feature extraction image data; (2) first edited image data; (3) second edited image data; 2 Either one of the edited image data and (4) the manually generated image data is acquired as map image data.

[0065] The generation step for inference generates inference processing result image data from map image data using a trained model of a trainable model obtained by a learning process using the learning processing method described in the sixth invention, or a trained model of a trainable model used in the generation step of the learning processing method described in the seventh invention obtained by a learning process using the learning processing method described in the seventh invention.

[0066] The label input unit inputs a label for the map image data.

[0067] The generated data acquisition process step acquires labeled image data that is a combination of the label input to the label input section and the inference processing result image data generated in the inference generating step.

[0068] This makes it possible to realize an image data generating method that has the same effects as the fourth aspect of the invention.

[0069] A ninth aspect of the present invention is a program for causing a computer to execute the learning data generating method according to the fifth aspect of the present invention.

[0070] This makes it possible to realize a program for causing a computer to execute a learning data generating method that has the same effects as the fifth aspect of the present invention.

[0071] A tenth aspect of the present invention is a program for causing a computer to execute the learning processing method according to the sixth or seventh aspect of the present invention.

[0072] This makes it possible to realize a program for causing a computer to execute a learning processing method that achieves the same effects as the sixth or seventh aspect of the present invention.

[0073] An eleventh aspect of the present invention is a program for causing a computer to execute the image data generating method according to the eighth aspect of the present invention.

[0074] This makes it possible to realize a program for causing a computer to execute an image data generating method that has the same effect as the eighth aspect of the present invention. Effect of the Invention

[0075] According to the present invention, it is possible to realize a medical diagnostic training data generation system, a training data generation device, a training processing device, an image data generation device, a training data generation method, a training processing method, an image data generation method, and a program that can collect large amounts of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning and perform accurate annotation. [Brief description of the drawings]

[0076] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical diagnosis learning data generation system 1000 according to a first embodiment. [Diagram 2] 1 is a diagram showing a schematic configuration of a training data generation device 100 of a medical diagnosis training data generation system 1000 according to a first embodiment. [Diagram 3] FIG. 1 is a diagram showing a schematic configuration of a learning processing device 200 of a medical diagnosis learning data generation system 1000 according to a first embodiment. [Figure 4] 1 is a diagram showing a schematic configuration of a medical image data generation device 300 of a medical diagnosis learning data generation system 1000 according to a first embodiment. [Diagram 5] FIG. 2 is a diagram for explaining the process (standardization process) of the standardization unit 12. [Figure 6] FIG. 2 is a diagram for explaining the process (standardization process) of the standardization unit 12. [Figure 7] 4 is a diagram for explaining the process (feature extraction process) of the feature extraction image generating unit 13. FIG. [Figure 8] 4 is a diagram for explaining the process (first edit process) of a first edit processing unit 14. FIG. [Figure 9] 5 is a diagram for explaining the process (second editing process) of the second edit processing unit 15. FIG. [Figure 10] A diagram showing the CPU bus configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0077] [First embodiment] The first embodiment will be described below with reference to the drawings.

[0078] <1.1: Configuration of medical diagnosis learning data generation system> FIG. 1 is a diagram showing a schematic configuration of a medical diagnosis learning data generation system 1000 according to the first embodiment.

[0079] FIG. 2 is a diagram showing a schematic configuration of the training data generation device 100 of the medical diagnosis training data generation system 1000 according to the first embodiment.

[0080] FIG. 3 is a diagram showing a schematic configuration of the learning processing device 200 of the medical diagnosis learning data generation system 1000 according to the first embodiment.

[0081] FIG. 4 is a diagram showing a schematic configuration of a medical image data generating device 300 of the medical diagnosis learning data generating system 1000 according to the first embodiment.

[0082] As shown in FIG. 1, the medical diagnosis learning data generation system 1000 includes a learning data generation device 100, a first data storage unit DB1, a learning processing device 200, a medical image data generation device 300, and a second data storage unit DB2.

[0083] (1.1.1: Learning data generating device 100) As shown in FIG. 2, the learning data generation device 100 includes an actual image input unit 11, a standardization unit 12, a feature extraction image generation unit 13, a first editing processing unit 14, a second editing processing unit 15, a manually generated image input unit 16, and a learning data generation unit 17.

[0084] The real image input unit 11 is a functional unit (e.g., a data input interface) for inputting actual medical image data (real medical image data). The real image input unit 11 outputs data including the input medical image data to the standardization unit 12 as data Din_real.

[0085] The standardization unit 12 inputs the data Din_real output from the real image input unit 11, acquires the medical image data (data of the real medical image) included in the data Din_real, and performs a standardization process on the medical image data (data of the real medical image).The standardization unit 12 then outputs data including the data after the standardization process (image data) as data D1_real to the feature extraction image generation unit 13, the second edit processing unit 15, and the learning data generation unit 17.

[0086] The feature extraction image generation unit 13 inputs the data D1_real output from the standardization unit 12, performs feature extraction image generation processing on the standardized image data included in the data D1_real, and generates (acquires) a feature extraction image through this processing. Then, the feature extraction image generation unit 13 outputs data including the feature extraction image generated (acquired) through the feature extraction image generation processing as data D1_map_auto to the first edit processing unit 14 and the learning data generation unit 17.

[0087] The first edit processing unit 14 inputs the data D1_map_auto output from the feature extraction image generation unit 13 and data D11_stylus indicating the editing contents (the data D11_stylus is, for example, data input from outside). The first edit processing unit 14 performs an edit process on the feature extraction image included in the data D1_map_auto based on the data D11_stylus indicating the editing contents, and obtains image data after the edit process. The first edit processing unit 14 then outputs data including the image data after the edit process to the learning data generation unit 17 as data D11_map_edited.

[0088] The second edit processing unit 15 receives data D1_real output from the standardization unit 12 and data D12_stylus (data D12_stylus is, for example, data input from outside) that indicates the edit content. The second edit processing unit 15 performs edit processing on the standardized image (image data) included in the data D1_real based on the data D12_stylus that indicates the edit content, and obtains the image data after the edit processing. The second edit processing unit 15 then outputs data including the image data after the edit processing to the learning data generation unit 17 as data D12_map_edited.

[0089] The manually generated image input unit 16 is a functional unit (e.g., a data input interface) for inputting manually generated image data. The manually generated image input unit 16 outputs data including the input image data to the learning data generation unit 17 as data D1_map_manual.

[0090] The learning data generation unit 17 receives as input the data D1_real output from the standardization unit 12, the data D1_map_auto output from the feature extraction image generation unit 13, the data D11_map_edited output from the first edit processing unit 14, the data D12_map_edited output from the second edit processing unit 15, and the data D1_map_manual output from the manually generated image input unit 16. The learning data generation unit 17 also receives as input a mode signal Mode. Note that the mode signal Mode is generated by a control unit (not shown) that controls the functional units of the learning data generation device 100, and is output from the control unit to the learning data generation unit 17.

[0091] The learning data generating unit 17 obtains learning data (learning data in which standardized real image data and map image data (condition image data) are paired) by pairing (1) the data D1_real output from the standardization unit 12 with any one of (2A) the data D1_map_auto output from the feature amount extraction image generating unit 13, (2B) the data D11_map_edited output from the first edit processing unit 14, (2C) the data D12_map_edited output from the second edit processing unit 15, and (2D) the data D1_map_manual output from the manually generated image input unit 16 in accordance with the mode signal Mode. Then, the learning data generating unit 17 outputs the obtained learning data as data Dtrain_pair (for example, to the first data storage unit DB1).

[0092] (1.1.2: First data storage unit DB1) The first data storage unit DB1 is a functional unit that can read data from a predetermined area (memory area) of the first data storage unit DB1 in response to an external data read command and / or can write data to a predetermined area (memory area) of the first data storage unit DB1 in response to an external data write command. The first data storage unit DB1 is realized by, for example, a data storage device, a database, or a memory device.

[0093] In accordance with a data write command from the device 100 for generating learning data, the first data storage unit DB1 writes the data Dtrain_pair output from the learning data generation unit 17 of the device 100 for generating learning data into a predetermined area (memory area) of the first data storage unit DB1.

[0094] In addition, in accordance with a data read command from the learning processing device 200, the first data storage unit DB1 reads out data stored in a specific area (memory area) of the first data storage unit DB1, and outputs the read out data to the learning processing device 200.

[0095] (1.1.3: Learning processing device 200) 3, the learning processing device 200 includes a data input unit 21, a generator 22, a first combination unit 23A, a second combination unit 23B, a selector SEL1, a downsampling unit 24, a first classifier 25A, a second classifier 25B, and a loss evaluation unit 26. The learning processing device 200 is a device for executing a learning process using a generative adversarial network model (GAN model (GAN: Generative Adversarial Network)) having a conditional input, that is, a so-called conditional GAN ​​model (learnable model).

[0096] The data input unit 21 outputs a data read command to the first data storage unit DB1, reads out learning data (learning data in which standardized actual image data and map image data (condition image data) are paired) stored in a predetermined area (memory area) of the first data storage unit DB1, and inputs the read learning data as data Din_pair (learning data Din_pair). The data input unit 21 extracts map image data (condition image data) from the learning data Din_pair, and outputs the extracted map image data (condition image data) as data Din_condition to the generator 22, the first combination unit 23A, and the second combination unit 23B.

[0097] Furthermore, the data input unit 21 extracts standardized real image data from the learning data Din_pair, and outputs the extracted standardized real image data to the second combining unit 23B as data Din_real.

[0098] The generator 22 is a generator of a conditional GAN ​​model and is a trainable model (the parameters of the generator are θ (G) The generator 22 inputs the map image data (condition image data) Din_condition output from the data input unit 21. The generator 22 inputs the map image data (condition image data) Din_condition to the trainable model of the generator 22, executes processing by the trainable model, and outputs the data acquired by the processing to the first combination unit 23A as data Do_G.

[0099] The first combination unit 23A inputs the data Do_G output from the generator 22 and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Do_G and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sG.

[0100] The second combining unit 23B receives the data Din_real and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Din_real and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sx.

[0101] The selector SEL1 is a two-input, one-output selector that selects either the data Do_G output from the first combination unit 23A or the data Do_sx output from the second combination unit 23B in accordance with a selection signal sel1 from a control unit (not shown) that controls the functional units of the learning processing device 200, and outputs the selected data as data Do0 to the downsampling unit 24 and the second discriminator 25B.

[0102] The downsampling unit 24 inputs data Do0 output from the selector SEL1, performs a downsampling process on the data Do0, obtains downsampled data, and outputs the obtained data to the first discriminator 25A as data Do1.

[0103] The first discriminator 25A is a discriminator of a conditional GAN ​​model, and is a trainable model (the parameters of the first discriminator are set to θ (D1) and includes T (T: natural number) layers (layers that extract features). The first classifier 25A inputs data Do1 output from the downsampling unit 24. The first classifier 25A inputs the data Do1 output from the downsampling unit 24 to a trainable model of the first classifier 25A, executes processing by the trainable model (processing for determining whether the input data Do1 is real / fake), and outputs data including the determination result data acquired by the processing and output data (feature data) of the T (T: natural number) layers (layers that extract features) to the loss evaluation unit 26 as data Do_D1.

[0104] The second discriminator 25B is a discriminator of a conditional GAN ​​model and is a trainable model (the parameters of the second discriminator are set as θ (D2) The second classifier 25B inputs data Do0 output from the selector SEL1. The second classifier 25B inputs the data Do0 output from the selector SEL1 to the trainable model of the second classifier 25B, executes processing by the trainable model (processing for determining whether the input data Do0 is real / fake), and outputs data including the determination result data acquired by the processing and output data (feature data) of the T layers (layers for extracting features) (T: natural number) to the loss evaluation unit 26 as data Do_D2.

[0105] The loss evaluation unit 26 receives the judgment result data Do_D1 output from the first classifier 25A, the judgment result data Do_D2 output from the second classifier 25B, and the teacher data D_correct, and acquires an error (loss acquired by a loss function) based on the judgment result data Do_D1 and the teacher data D_correct, and an error (loss acquired by a loss function) based on the judgment result data Do_D2 and the teacher data D_correct. Then, the loss evaluation unit 26 performs parameter update processing (learning processing) of the trainable models of the first classifier 25A, the second classifier 25B, and the generator 22 based on the acquired error (loss) (for example, performs learning processing by an error backpropagation method). In addition, the teacher label (corresponding to the teacher data D_correct) for the judgment data as to whether it is real or fake, contained in the output data Dout_D1, Dout_D2 of the trainable model of the first classifier 25A and the second classifier 25B, is “1” indicating that the data is real when data including real data (including data after downsampling) is input to the first classifier 25A and the second classifier 25B, and is “0” indicating that the data is fake when data including fake data (output data of the generator 22) (including data after downsampling) is input to the first classifier 25A and the second classifier 25B.

[0106] In addition, the loss evaluation unit 26 inputs the data Din_condition (the probability distribution of this data is set to "s") output from the data input unit 21, the data Do_G (the probability distribution of this data is set to "G(s)") output from the generator 22, and the data Din_real (the probability distribution of this data is set to "x") output from the data input unit 21, and evaluates the loss of the conditional GAN ​​model using a loss function based on the input data, and performs a process of determining whether the learning process has converged.

[0107] (1.1.4: Medical image data generating device 300) As shown in Figure 4, the medical image data generation device 300 includes a second actual image input unit 30, a second standardization unit 31, a second feature extraction image generation unit 32, a third editing processing unit 33, a fourth editing processing unit 34, a second manually generated image input unit 35, a map image generation unit 36, a label input unit 37, a generator 38, and a generated data acquisition processing unit 39.

[0108] The second real image input unit 30 is a functional unit (e.g., a data input interface) for inputting actual medical image data (real medical image data). The second real image input unit 30 outputs data including the input medical image data to the second standardization unit 31 as data Din2_real.

[0109] The second standardization unit 31 inputs the data Din2_real output from the second actual image input unit 30, acquires the medical image data (data of the actual medical image) included in the data Din2_real, and performs a standardization process on the medical image data (data of the actual medical image).The second standardization unit 31 then outputs data including the data after the standardization process (image data) as data D2_real to the second feature extraction image generation unit 32 and the third edit processing unit 33.

[0110] The second feature extraction image generation unit 32 inputs the data D2_real output from the second standardization unit 31, performs feature extraction image generation processing on the standardized image data included in the data D2_real, and generates (acquires) a feature extraction image through this processing. The second feature extraction image generation unit 32 then outputs data including the feature extraction image generated (acquired) through the feature extraction image generation processing to the third edit processing unit 33 as data D2_map_auto.

[0111] The third edit processing unit 33 receives data D2_map_auto output from the second feature extraction image generating unit 32 and data D21_stylus indicating the editing content (data D21_stylus is, for example, data input from outside). The third edit processing unit 33 performs an edit process on the feature extraction image included in data D21_map_auto based on the data D21_stylus indicating the editing content, and obtains image data after the edit process. The third edit processing unit 33 then outputs data including the image data after the edit process to the map image generating unit 36 ​​as data D21_map_edited.

[0112] The fourth edit processor 34 receives the data D2_real output from the second standardization unit 31 and data D22_stylus (the data D22_stylus is, for example, data input from an external source) that indicates the editing content. The fourth edit processor 34 performs an edit process on the standardized image (image data) included in the data D2_real based on the data D22_stylus that indicates the editing content, and obtains the image data after the edit process. The fourth edit processor 34 then outputs data including the image data after the edit process to the map image generator 36 as data D22_map_edited.

[0113] The second manually generated image input unit 35 is a functional unit (e.g., a data input interface) for inputting manually generated image data. The second manually generated image input unit 35 outputs data including the input image data to the map image generation unit 36 ​​as data D2_map_manual.

[0114] The map image generating unit 36 ​​inputs the data D2_map_auto output from the second feature extraction image generating unit 32, the data D21_map_edited output from the third edit processing unit 33, the data D22_map_edited output from the fourth edit processing unit 34, and the data D2_map_manual output from the second manually generated image input unit 35. The map image generating unit 36 ​​also inputs a mode signal Mode. The mode signal Mode is generated by a control unit (not shown) that controls the functional units of the medical image data generating device 300, and is output from the control unit to the map image generating unit 36.

[0115] The map image generation unit 36 ​​selects one of the following data in accordance with the mode signal Mode: (1) data D2_map_auto output from the second feature extraction image generation unit 32, (2) data D21_map_edited output from the third edit processing unit 33, (3) data D22_map_edited output from the fourth edit processing unit 34, and (4) data D2_map_manual output from the second manually generated image input unit 35, and outputs the selected data to the generator 38 as data Do_map.

[0116] Label input unit 37 is a functional unit (e.g., a data input interface) for inputting data (information) for identifying a label corresponding to the map image output from map image generation unit 36. Label input unit 37 outputs the input data (label data) to generated data acquisition processing unit 39 as data Lbl.

[0117] The generator 38 is a generator of a conditional GAN ​​model, and has the same function and configuration as the generator 22 of the learning processing device 200. The generator 38 includes a trained model (a model of a generator that has completed the learning process by the learning processing device 200 (optimal parameters are θ opt (G) The generator 38 is equipped with a map image data (condition image data) Do_map that is set. The generator 38 inputs the map image data (condition image data) Do_map that is output from the map image generation unit 36. The generator 38 inputs the map image data (condition image data) Do_map to the trained model of the generator 38, executes processing by the trained model, and outputs data acquired by the processing to the generated data acquisition processing unit 39 as data D_gene_out.

[0118] The generated data acquisition processing unit 39 inputs the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37. The generated data acquisition processing unit 39 performs a process (generated data acquisition process) of making the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37 into a set of data (pair data (labeled data)), and outputs the data acquired by this process to the second data storage unit DB2 as data Dout (={D_gene_out, Lbl}).

[0119] (1.1.5: Second data storage unit DB2) The second data storage unit DB2 is a functional unit that can read data from a predetermined area (memory area) of the second data storage unit DB2 in response to an external data read command and / or can write data to a predetermined area (memory area) of the second data storage unit DB2 in response to an external data write command. The second data storage unit DB2 is realized by, for example, a data storage device, a database, or a memory device.

[0120] In accordance with a data write command from the medical image data generation device 300, the second data storage unit DB2 writes the data Dout output from the generated data acquisition processing unit 39 of the medical image data generation device 300 into a specified area (memory area) of the second data storage unit DB2.

[0121] <1.2: Operation of the medical diagnosis learning data generation system> The operation of the medical diagnosis learning data generation system 1000 configured as above will be described with reference to the drawings.

[0122] In the following, the operation of the medical diagnosis training data generation system 1000 will be described separately for (1) training data generation processing, (2) training processing, and (3) medical image data generation processing.

[0123] 5 and 6 are diagrams for explaining the process (standardization process) of the standardization unit 12. FIG.

[0124] FIG. 7 is a diagram for explaining the process (feature extraction process) of the feature extraction image generating unit 13. As shown in FIG.

[0125] FIG. 8 is a diagram for explaining the process (first edit process) of the first edit processing unit 14. As shown in FIG.

[0126] FIG. 9 is a diagram for explaining the process (second edit process) of the second edit processing unit 15. As shown in FIG.

[0127] (1.2.1: Learning data generation process) First, the learning data generation process will be described.

[0128] Actual medical image data (real medical image data) is input to the real image input unit 11 (for example, a user inputs real medical image data from outside to the real image input unit 11). The real image input unit 11 outputs data including the input medical image data to the standardization unit 12 as data Din_real.

[0129] The standardization unit 12 inputs the data Din_real output from the real image input unit 11, acquires medical image data (data of the real medical image) included in the data Din_real, and performs standardization processing on the medical image data (data of the real medical image). For example, the standardization unit 12 performs standardization processing on the medical image data (data of the real medical image) by performing (1) a process of cutting out (extracting) a predetermined region (image region) of the image (for example, a process of cutting out (extracting) the left and right regions (or the top and bottom regions) of the image), (2) a process of padding the predetermined region of the image (padding process), and (3) a process of changing the size of the image, and acquires a standardized image (for example, an image of a predetermined size) through the standardization processing.

[0130] For example, when the image shown in FIG. 5 is input to the standardization unit 12 as medical image data (data of an actual medical image), the standardization unit 12 can obtain an image (the image shown on the right side of FIG. 5) obtained by cutting out the left and right image regions of the input medical image data (data of an actual medical image) as a standard image (for example, an image with an image size of x1 × y1) by performing (1) a process of cutting out the left and right regions of the image, (2) a padding process (a process of padding the left and right image regions with zeros after the cut-out process), and (3) a resizing process.

[0131] In addition, when the image shown in Figure 6 is input to the standardization unit 12 as medical image data (data of an actual medical image), the standardization unit 12 can obtain an image (the image shown on the right side of Figure 6) cut out from a specified image area of ​​the input medical image data (data of an actual medical image) as a standard image (for example, an image with an image size of x1 x y1) by performing (1) padding processing (processing of padding the left and right image areas with zeros after the cut-out processing) and (2) resizing processing.

[0132] The standardization unit 12 outputs data including the standardized data (image data) to the feature extraction image generation unit 13, the second edit processing unit 15, and the learning data generation unit 17 as data D1_real.

[0133] The feature extraction image generation unit 13 inputs the data D1_real output from the standardization unit 12, and performs a feature extraction image generation process on the image data after standardization process contained in the data D1_real, thereby generating (obtaining) a feature extraction image.

[0134] For example, when the data (standardized image) D1_real after standardization shown in Fig. 7 is input to the feature extraction image generation unit 13, the feature extraction image generation unit 13 executes feature extraction image generation processing (e.g., edge extraction processing) on ​​the input standard image, and obtains an image from which features have been extracted (e.g., an edge extraction image) (e.g., the image shown on the right side of Fig. 7). Note that in Fig. 7, the feature extraction image (e.g., the edge extraction image) is displayed with negative / positive inversion.

[0135] Then, the feature extraction image generating unit 13 outputs data including the feature extraction image generated (acquired) by the feature extraction image generating process to the first edit processing unit 14 and the learning data generating unit 17 as data D1_map_auto.

[0136] The first edit processing unit 14 receives data D1_map_auto output from the feature extraction image generating unit 13 and data D11_stylus instructing the editing contents (the data D11_stylus is, for example, data input from an external source). The first edit processing unit 14 performs an edit process on the feature extraction image included in the data D1_map_auto based on the data D11_stylus instructing the editing contents, and obtains image data after the edit process.

[0137] For example, when the feature extraction image data (for example, edge extraction image) D1_map_auto shown in FIG. 8 is input to the first edit processing unit 14, the first edit processing unit 14 displays the feature extraction image data (for example, edge extraction image) D1_map_auto on, for example, a touch panel display device (not shown), and performs an edit process on the feature extraction image data (for example, edge extraction image) D1_map_auto by, for example, having the user draw or erase in a predetermined image area on the displayed image with a touch pen. Then, the data of the edit process is input to the first edit processing unit 14 as data D11_stylus. The first edit processing unit 14 performs an edit process on the feature extraction image data (for example, edge extraction image) D1_map_auto based on the data D11_stylus instructing the edit contents, and obtains data of the image after the edit process (for example, the image shown on the right side of FIG. 8). Note that in FIG. 8, the feature extraction image (for example, edge extraction image) and the image after the edit process are displayed with negative / positive inversion.

[0138] Then, the first edit processing unit 14 outputs data including the image data after the edit processing to the learning data generation unit 17 as data D11_map_edited.

[0139] The second edit processing unit 15 receives data D1_real of the standardized image output from the standardization unit 12 and data D12_stylus indicating the editing contents (the data D12_stylus is, for example, data input from outside). The second edit processing unit 15 performs an edit process on the standardized image included in the standardized image data D1_real based on the data D12_stylus indicating the editing contents, and obtains the image data after the edit process.

[0140] For example, when the standardized image data D1_real shown in FIG. 9 is input to the second edit processing unit 15, the second edit processing unit 15 displays the standardized image data D1_real on, for example, a touch panel display device (not shown), and performs an edit process on the standardized image data D1_real (for example, a process of acquiring image data created by drawing by tracing the edge parts of the standardized image) by the user using, for example, a touch pen in a predetermined image area on the displayed image. Then, data of the edit process is input to the second edit processing unit 15 as data D12_stylus. The second edit processing unit 15 performs an edit process on the standardized image data D1_real based on data D12_stylus instructing the editing contents, and acquires data of the image after the edit process (for example, the image shown on the right side of FIG. 9). Note that the image on the right side of FIG. 9 is displayed with negative / positive inversion.

[0141] Then, the second edit processing unit 15 outputs data including the image data after the edit processing to the learning data generating unit 17 as data D12_map_edited.

[0142] The manually generated image input unit 16 is a functional unit (e.g., a data input interface) for inputting manually generated image data, and outputs data including the input image data as data D1_map_manual to the learning data generation unit 17. For example, the manually generated image input unit 16 has a touch panel display device (not shown), and on the touch panel display device, a user draws and creates an image similar to the feature extraction image data (e.g., edge extraction image) D1_map_auto, for example, using a touch pen. Then, the manually generated image input unit 16 outputs data including the input image data (an image manually created by the user (an image similar to the feature extraction image data (e.g., edge extraction image) D1_map_auto)) to the learning data generation unit 17 as data D1_map_manual.

[0143] The learning data generation unit 17 inputs the data D1_real output from the standardization unit 12, the data D1_map_auto output from the feature extraction image generation unit 13, the data D11_map_edited output from the first edit processing unit 14, the data D12_map_edited output from the second edit processing unit 15, and the data D1_map_manual output from the manually generated image input unit 16. The learning data generation unit 17 also inputs a mode signal Mode from a control unit (not shown).

[0144] The learning data generation unit 17, in accordance with the mode signal Mode, pairs (1) the data D1_real output from the standardization unit 12 with one of the following data: (2A) the data D1_map_auto output from the feature extraction image generation unit 13, (2B) the data D11_map_edited output from the first editing processing unit 14, (2C) the data D12_map_edited output from the second editing processing unit 15, and (2D) the data D1_map_manual output from the manually generated image input unit 16, thereby obtaining learning data (learning data in which standardized real image data is paired with map image data (condition image data)).

[0145] For example, the learning data generating unit 17 performs the following process in accordance with the signal value of the mode signal Mode. (1) When the signal value of the mode signal Mode is “1”, the training data generation unit 17 acquires, as training data Dtrain_pair, data (={D1_real, D1_map_auto}) that is a pair of data D1_real output from the standardization unit 12 and data D1_map_auto output from the feature extraction image generation unit 13. (2) When the signal value of the mode signal Mode is “2”, the learning data generation unit 17 acquires the learning data Dtrain_pair, which is a pair of data D1_real output from the standardization unit 12 and data D11_map_edited output from the first editing processing unit 14 (={D1_real, D11_map_edited}). (3) When the signal value of the mode signal Mode is “3”, the learning data generation unit 17 acquires the data (={D1_real, D12_map_edited}) that is a pair of data D1_real output from the standardization unit 12 and data D12_map_edited output from the second editing processing unit 15 as learning data Dtrain_pair. (4) When the signal value of the mode signal Mode is “4”, the learning data generation unit 17 acquires data (={D1_real, D1_map_manual}) that is a pair of data D1_real output from the standardization unit 12 and data D1_map_manual output from the manually generated image input unit 16 as learning data Dtrain_pair.

[0146] Then, the learning data generating unit 17 outputs the acquired learning data Dtrain_pair to the first data storage unit DB1.

[0147] Then, in accordance with a data write command from the learning data generation device 100, the first data storage unit DB1 writes the data Dtrain_pair output from the learning data generation unit 17 of the learning data generation device 100 into a predetermined area (memory area) of the first data storage unit DB1.

[0148] (1.2.2: Learning process) Next, the learning process will be described.

[0149] The data input unit 21 of the learning processing device 200 outputs a data read command to the first data storage unit DB1, reads out learning data (learning data in which standardized actual image data and map image data (condition image data) are paired) stored in a predetermined area (memory area) of the first data storage unit DB1, and inputs the read learning data as data Din_pair (learning data Din_pair). The data input unit 21 extracts map image data (condition image data) from the learning data Din_pair, and outputs the extracted map image data (condition image data) as data Din_condition to the generator 22, the first combination unit 23A, and the second combination unit 23B.

[0150] Furthermore, the data input unit 21 extracts standardized real image data from the learning data Din_pair, and outputs the extracted standardized real image data to the second combining unit 23B as data Din_real.

[0151] Generator 22, which is the generator of the conditional GAN ​​model (the generator parameters are θ (G) The data input unit 23A inputs map image data (condition image data) Din_condition output from the data input unit 21. The generator 22 inputs the map image data (condition image data) Din_condition to the trainable model of the generator 22, executes processing using the trainable model, and outputs the data acquired by the processing to the first combination unit 23A as data Do_G.

[0152] The first combination unit 23A inputs the data Do_G output from the generator 22 and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Do_G and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sG.

[0153] The second combining unit 23B receives the data Din_real and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Din_real and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sx.

[0154] The selector SEL1 selects either the data Do_sG output from the first combining unit 23A or the data Ds_sx output from the second combining unit 23B in accordance with a selection signal sel1 from a control unit (not shown) that controls the functional units of the learning processing device 200. When the signal value of the selection signal sel is "0", the selector SEL1 selects the data Do_G (data input to terminal 0) output from the generator 22, and outputs the data Do_G as data Do0 to the downsampling unit 24 and the second discriminator 25B. On the other hand, when the signal value of the selection signal sel is "1", the selector SEL1 selects the data Do_sx output from the second combining unit 23B (data input to terminal 1), and outputs the data Do_sx as data Do0 to the downsampling unit 24 and the second discriminator 25B.

[0155] The downsampling unit 24 inputs data Do0 output from the selector SEL1, performs a downsampling process on the data Do0, obtains downsampled data, and outputs the obtained data to the first discriminator 25A as data Do1.

[0156] The first discriminator 25A, which is a discriminator of the conditional GAN ​​model (the parameters of the discriminator are θ (D1)The first classifier 25A inputs the data Do1 output from the downsampling unit 24 to the trainable model of the first classifier 25A, executes processing by the trainable model (processing for determining whether the input data Do1 is real / fake), and outputs data including the determination result data acquired by the processing and the output data (feature data) of the T layers (layers for extracting features) as data Do_D1 to the loss evaluation unit 26. Note that the trainable model of the first classifier 25A outputs "1" when it determines that the input data Do1 is real, and outputs "0" when it determines that the input data D2 is fake.

[0157] A second discriminator 25B (with the parameters of the discriminator set to θ (D2) The second classifier 25B inputs the data Do0 output from the selector SEL1 to the trainable model of the second classifier 25B, executes processing by the trainable model (processing for determining whether the input data Do0 is real / fake), and outputs data including the determination result data acquired by the processing and the output data (feature data) of the T layers (layers for extracting features) as data Do_D2 to the loss evaluation unit 26. Note that the trainable model of the second classifier 25B outputs "1" when it determines that the input data Do1 is real, and outputs "0" when it determines that the input data D2 is fake.

[0158] In addition, the teacher label (corresponding to the teacher data D_correct) for the judgment data as to whether it is real or fake, contained in the output data Dout_D1, Dout_D2 of the trainable model of the first classifier 25A and the second classifier 25B, is “1” indicating that the data is real when data including real data (including data after downsampling) is input to the first classifier 25A and the second classifier 25B, and is “0” indicating that the data is fake when data including fake data (output data of the generator 22) (including data after downsampling) is input to the first classifier 25A and the second classifier 25B.

[0159] The loss evaluation unit 26 acquires the determination result data Do_D1 output from the first classifier 25A, the determination result data Do_D2 output from the second classifier 25B, an error (loss acquired by a loss function) based on the determination result data Do_D1 and the teacher data D_correct, and an error (loss acquired by a loss function) based on the determination result data Do_D2 and the teacher data D_correct. Then, the loss evaluation unit 26 performs parameter update processing (learning processing) of the trainable models of the first classifier 25A, the second classifier 25B, and the generator 22 based on the acquired error (loss) (for example, performs learning processing by an error backpropagation method).

[0160] Specifically, the learning processing device 200 performs the learning process by carrying out processing corresponding to the following formulas.

number

number

[0161] In addition, a feature matching loss function LFM(G, D) for the generator G (the generator 22) and the k-th classifier (the first classifier 25A and the second classifier 25B) is k ) is as follows:

number

[0162] Then, (1) the loss function L for the generator G (the generator 22) and the classifier D (the first classifier 25A and the second classifier 25B) obtained as above is GAN (G,D k ), and (2) a feature matching loss function L for the generator G (the generator 22) and the k-th classifier (the first classifier 25A and the second classifier 25B). FM (G,D k ) in the learning processing device 200, a process corresponding to (Equation 1) is performed to obtain the parameter θ (G) The update process of the parameter θ of the first classifier 25A (D1) and the update process of the parameter θ (D2) The parameter θ (G) The optimal parameter θ opt (G) , the parameter θ of the first classifier 25A (D1) The optimal parameter θ opt (D1) , and the parameter θ of the second classifier 25B (D2) The optimal parameter θ opt (D2) In the learning processing device 200, the parameter θ (G) The optimal parameter θ opt (G) , the parameter θ of the first classifier 25A (D1) The optimal parameter θ opt (D1) , and the parameter θ of the second classifier 25B (D2) The optimal parameter θopt (D2) Once acquired, the learning process is terminated.

[0163] (1.2.3: Medical image data generation processing) Next, the medical image data generation process will be described.

[0164] The generator 38 of the medical image data generating device 300 is provided with the optimal parameters θ of the generator acquired by the learning processing of the learning processing device 200. opt (G) It is equipped with a model (pre-trained model) with the following settings:

[0165] Actual medical image data (real medical image data) is input to the second real image input unit 30 (for example, a user inputs real medical image data from outside to the second real image input unit 30). The second real image input unit 30 outputs data including the input medical image data to the second standardization unit 31 as data Din2_real.

[0166] The second standardization unit 31 inputs the data Din2_real output from the second actual image input unit 30, acquires the medical image data (data of the actual medical image) included in the data Din2_real, and performs standardization processing on the medical image data (data of the actual medical image). For example, the second standardization unit 31 performs standardization processing on the medical image data (data of the actual medical image) by performing (1) a process of cutting out (extracting) a predetermined region (image region) of the image (for example, a process of cutting out (extracting) the left and right regions (or the top and bottom regions) of the image), (2) a process of padding the predetermined region of the image (padding processing), and (3) a process of changing the size of the image, and acquires a standardized image (for example, an image of a predetermined size) through the standardization processing (performing the same processing as the standardization unit 12).

[0167] Then, the second standardization unit 31 outputs data including the data (image data) after the standardization process as data D2_real to the second feature amount extraction image generation unit 32 and the fourth edit processing unit .

[0168] The second feature extraction image generation unit 32 inputs the data D2_real output from the second standardization unit 31, performs a feature extraction image generation process on the image data after standardization process contained in the data D2_real, and generates (obtains) a feature extraction image through this process (performs the same process as the feature extraction image generation unit 13).

[0169] Then, the second feature extraction image generating unit 32 outputs data including the feature extraction image generated (acquired) by the feature extraction image generating process to the third edit processing unit 33 and the map image generating unit 36 ​​as data D2_map_auto.

[0170] The third edit processing unit 33 receives data D2_map_auto output from the second feature extraction image generating unit 32 and data D21_stylus indicating the editing contents (the data D21_stylus is, for example, data input from an external source). The third edit processing unit 33 performs an edit process on the feature extraction image included in the data D2_map_auto based on the data D21_stylus indicating the editing contents, and obtains image data after the edit process (performing the same process as the first edit processing unit 14).

[0171] Then, the third edit processing unit 33 outputs data including the image data after the edit processing to the map image generating unit 36 ​​as data D21_map_edited.

[0172] The fourth edit processing unit 34 receives data D2_real of the standardized image output from the second standardization unit 31 and data D22_stylus indicating the editing contents (the data D22_stylus is, for example, data input from an external source). The fourth edit processing unit 34 performs an edit process on the standardized image included in the standardized image data D2_real based on the data D22_stylus indicating the editing contents, and obtains the image data after the edit process (performing the same process as the second edit processing unit 15).

[0173] Then, the fourth edit processing unit 34 outputs data including the image data after the edit processing to the map image generating unit 36 ​​as data D22_map_edited.

[0174] The second manually generated image input unit 35 is a functional unit (e.g., a data input interface) for inputting manually generated image data, and outputs data including the input image data to the map image generating unit 36 ​​as data D2_map_manual. For example, the second manually generated image input unit 35 has a touch panel display device (not shown), and on the touch panel display device, the user draws and creates an image similar to the feature extraction image data (e.g., edge extraction image) D2_map_auto, for example, using a touch pen. Then, the second manually generated image input unit 35 outputs data including the input image data (an image manually created by the user (an image similar to the feature extraction image data (e.g., edge extraction image) D2_map_auto)) to the map image generating unit 36 ​​as data D2_map_manual.

[0175] The map image generation unit 36 ​​receives as input the data D2_map_auto output from the second feature extraction image generation unit 32, the data D21_map_edited output from the third edit processing unit 33, the data D22_map_edited output from the fourth edit processing unit 34, and the data D2_map_manual output from the second manually generated image input unit 35. The map image generation unit 36 ​​also receives as input a mode signal Mode from a control unit (not shown).

[0176] In accordance with the mode signal Mode, the map image generation unit 36 ​​acquires one of the following data: (1) data D2_map_auto output from the second feature extraction image generation unit 32, (2) data D21_map_edited output from the third editing processing unit 33, (3) data D22_map_edited output from the fourth editing processing unit 34, and (4) data D2_map_manual output from the second manually generated image input unit 35, as map image data Do_map to be output to the generator 38.

[0177] For example, the learning data generating unit 17 performs the following process in accordance with the signal value of the mode signal Mode. (1) When the signal value of the mode signal Mode is "1", the map image generating unit 36 ​​acquires the data D2_map_auto output from the second feature amount extraction image generating unit 32 as map image data Do_map. (2) When the signal value of the mode signal Mode is "2", the map image generating unit 36 ​​obtains the data D21_map_edited output from the third edit processing unit 33 as the map image data Do_map. (3) When the signal value of the mode signal Mode is "3", the map image generation unit 36 ​​obtains the data D22_map_edited output from the fourth edit processing unit 34 as the map image data Do_map. (4) When the signal value of the mode signal Mode is "4", the map image generating unit 36 ​​acquires the data D2_map_manual output from the second manually generated image input unit 35 as the map image data Do_map.

[0178] Then, map image generating unit 36 ​​outputs the map image data Do_map obtained as described above to generator 38.

[0179] The generator 38 is provided with a trained model (a model of a generator that has completed training processing by the training processing device 200 (optimum parameters are set to θ opt (G) The generator 38 is equipped with a map image data (condition image data) Do_map that is set. The generator 38 inputs the map image data (condition image data) Do_map that is output from the map image generation unit 36. The generator 38 inputs the map image data (condition image data) Do_map to the trained model of the generator 38, executes processing by the trained model, and outputs data acquired by the processing to the generated data acquisition processing unit 39 as data D_gene_out.

[0180] Data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 ​​is input to the label input unit 37. Then, the label input unit 37 outputs the input data (label data) to the generated data acquisition processing unit 39 as data Lbl. Note that since the user knows in advance what kind of data the actual image data input to the second actual image input unit 30 or the data input to the second manually generated image input unit 35 is (since the user knows in advance that the actual image data input to the second actual image input unit 30 or the data input to the second manually generated image input unit 35 is, for example, medical image data of a person suffering from a specific disease or medical image data of a healthy state), the data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 ​​can be accurately input to the label input unit 37.

[0181] The generated data acquisition processing unit 39 inputs the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37. The generated data acquisition processing unit 39 performs a process (generated data acquisition process) of making the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37 into a set of data (pair data (labeled data)), and outputs the data acquired by this process to the second data storage unit DB2 as data Dout (={D_gene_out, Lbl}).

[0182] By repeatedly executing the above process in the medical image data generation device 300, a large amount of highly accurate labeled medical image data can be generated.

[0183] <Summary> As described above, in the medical diagnosis training data generation system 1000, the training data generation device 100 generates various training image data, i.e., (1) Learning data (learning data in the automatic generation mode) consisting of a combination of standardized actual image data and map image (condition image) data (image data from which features have been extracted) automatically generated from the actual image data; (2) Learning data consisting of a combination of standardized actual image data and map image (condition image) data obtained by editing a map image automatically generated from the actual image data (learning data in semi-automatic mode (map image editing mode)); (3) Learning data consisting of a combination of standardized actual image data and map image (condition image) data obtained by performing editing processing based on the actual image data (learning data in semi-automatic mode (actual image editing mode)); (4) Training data (training data in manual mode) consisting of standardized actual image data and manually generated map image (condition image) data; can be acquired. In the medical diagnosis training data generation system 1000, the training processing device 200 can train a trainable model (for example, a conditional GAN ​​model) using various training image data acquired by the training data generation device 100. In other words, in the training process of the medical diagnosis training data generation system 1000, the training processing device 200 can train a trainable model (for example, a conditional GAN ​​model) using real image data that can only be acquired in small quantities and map image data acquired based on the real image data. Therefore, even if only a small amount of real image data can be acquired, a large amount of training data for training the trainable model can be acquired. As a result, the training processing device 200 can train a trainable model (for example, a conditional GAN ​​model) with high accuracy.

[0184] In the medical diagnosis learning data generation system 1000, the medical image data generation device 300 generates various input image data (input data for inference processing) based on a small number of actual image data, i.e., (1) Map image (condition image) data (image data with extracted features) automatically generated from standardized real image data (input data for inference using the automatic generation mode), (2) Map image (condition image) data obtained by editing a map image automatically generated from standardized real image data (input data for inference using semi-automatic mode (map image editing mode)), (3) Map image (condition image) data obtained by editing the standardized real image data (input data for inference in semi-automatic mode (real image editing mode)), (4) Manually generated map image (condition image) data (input data for inference in manual mode), can be obtained. In the medical diagnosis learning data generation system 1000, the various input data for inference (image data) obtained as described above are input to a generator 38 equipped with a trained model, whereby highly accurate generated image data D_gene_out can be obtained. Furthermore, in the medical diagnosis learning data generation system 1000, the user knows in advance what kind of data the real image data input to the second real image input unit 30 of the medical image data generation device 300 or the data input to the second manually generated image input unit 35 is (the user knows in advance that the real image data input to the second real image input unit 30 or the data input to the second manually generated image input unit 35 is, for example, medical image data of a subject suffering from a specific disease or medical image data of a healthy state, etc.), so that data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 ​​can be accurately input to the label input unit 37. The generated data acquisition processing unit 39 of the medical image data generation device 300 can acquire data (paired data (labeled data)) Dout that is a pair of data D_gene_out output from the generator 38 and data Lbl output from the label input unit 37, so that accurately annotated data (labeled data) can be easily acquired in large quantities.

[0185] As described above, the medical diagnostic training data generation system 1000 can collect a large amount of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation.

[0186] [Other embodiments] In the above embodiment, a case where a conditional GAN ​​model is adopted in the learning processing device 200 of the medical diagnosis learning data generation system 1000 has been described, but the present invention is not limited to this, and other generation models that can be input with conditions (for example, a diffusion model that can be input with conditions) may be adopted. In other words, the learning processing device 200 may perform a learning process using other generation models that can be input with conditions (for example, a diffusion model that can be input with conditions), and the trained model acquired by the learning process may be loaded into the medical image data generation device 300, and the medical image data generation device 300 may perform an inference process (a process of generating medical image data).

[0187] In the medical diagnosis learning data generation system 1000 described in the above embodiment, each block (each functional unit) may be individually integrated into one chip using a semiconductor device such as an LSI, or may be integrated into one chip to include some or all of the blocks. In addition, each block (each functional unit) of the medical diagnosis learning data generation system 1000, the learning data generation device 100, the learning processing device 200, and the medical image data generation device 300 described in the above embodiment may be realized by multiple semiconductor devices such as LSIs.

[0188] Although we refer to it as LSI here, it may also be called IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0189] The method of integration is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. It is also possible to use a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI.

[0190] In addition, a part or all of the processing of each functional block in each of the above embodiments may be realized by a program. And a part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Also, the programs for performing each processing are stored in a storage device such as a hard disk or a ROM, and are executed in the ROM or by being read into the RAM.

[0191] In addition, each process in the above embodiment may be realized by hardware, or may be realized by software (including the case where it is realized together with an OS (operating system), middleware, or a predetermined library). Furthermore, it may be realized by a mixed process of software and hardware.

[0192] When each functional unit of the above embodiment is realized by software, for example, each functional unit may be realized by software processing using the hardware configuration shown in FIG. 10 (for example, a hardware configuration in which a CPU (which may be a GPU), a ROM, a RAM, an input unit, an output unit, a communication unit, a memory unit (for example, a memory unit realized by an HDD, an SSD, etc.), an external media drive, etc. are connected via a bus).

[0193] Furthermore, when each functional unit of the above embodiment is realized by software, the software may be realized using a single computer having the hardware configuration shown in FIG. 10, or may be realized by distributed processing using multiple computers.

[0194] In addition, the execution order of the processing method in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed without departing from the scope of the invention. In addition, in the processing method in the above embodiment, some steps may be executed in parallel with other steps without departing from the scope of the invention.

[0195] The scope of the present invention includes a computer program for causing a computer to execute the above-mentioned method, and a computer-readable recording medium having the program recorded thereon. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories.

[0196] The computer program is not limited to one recorded on the recording medium, but may be one transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, or the like.

[0197] The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention. [Explanation of symbols]

[0198] 1000 Medical diagnosis learning data generation system 100 Learning data generation device 200 Learning processing device 300 Medical image data generating device 11 Real image input unit 12 Standardization Department 13 Feature extraction image generation unit 14 First editing processing section 15 Second editing processing section 16 Manually generated image input section 17 Learning data generation unit 21 Data Entry Section 22 Generator 25A 1st discriminator 25B 1st discriminator 26 Loss Assessment Department 30 Second actual image input unit 31 2nd Standardization Department 32 Second feature extraction image generation unit 33 Third Editing Processing Section 34 4th Editing Processing Unit 35 Second manually generated image input unit 36 Map image generator 37 Label input section 38 Generator (inference generator) 39 Generated data acquisition processing unit

Claims

1. an image input unit for inputting image data; a standardization unit that performs a standardization process on the image data input to the image input unit to obtain standardized image data; a feature extraction image generating unit that performs a feature extraction process on the standardized image data to obtain feature extraction image data; a first edit processing unit that can input first edit data, which is data for editing the feature amount extracted image data, performs an edit process on the feature amount extracted image data based on the first edit data, and acquires the image data after the edit process as first edited image data; a second edit processing unit capable of inputting second edit data, which is data for editing the standardized image data, performing an edit process on the standardized image data based on the second edit data, and acquiring the image data after the edit process as second edited image data; a manually generated image input unit for inputting manually generated image data which is data of a manually generated map image; a learning data generating unit that acquires learning data that is a combination of the standardized image data and any one of image data of (1) the feature amount extraction image data, (2) the first edited image data, (3) the second edited image data, and (4) the manually generated image data; A learning data generating device comprising:

2. A learning processing device for performing learning processing using the learning data generated by the learning data generating device according to claim 1, a data input unit that inputs the learning data generated by the learning data generating device according to claim 1 and extracts condition image data and actual image data from the learning data; A learning processing unit including a trainable model, inputting the condition image data and the actual image data, and performing a learning process for the trainable model; a loss evaluation unit that evaluates an error between the data output from the learning processing unit and training data; Equipped with Learning processing device.

3. The learning processing unit: A generator including a trainable model that receives the condition image data and generates image data based on the received condition image data; a classifier that receives the condition image data and either the image data generated by the generator or the actual image data, and outputs a determination process result as to whether the input data is real data or fake data; Equipped with The learning processing device according to claim 2 .

4. an image input unit for inputting image data; a standardization unit that performs a standardization process on the image data input to the image input unit to obtain standardized image data; a feature extraction image generating unit that performs a feature extraction process on the standardized image data to obtain feature extraction image data; a first edit processing unit that can input first edit data, which is data for editing the feature amount extracted image data, performs an edit process on the feature amount extracted image data based on the first edit data, and acquires the image data after the edit process as first edited image data; a second edit processing unit capable of inputting second edit data, which is data for editing the standardized image data, performing an edit process on the standardized image data based on the second edit data, and acquiring the image data after the edit process as second edited image data; a manually generated image input unit for inputting manually generated image data which is data of a manually generated map image; a map image generating unit that acquires, as map image data, any one of image data of (1) the feature extraction image data, (2) the first edited image data, (3) the second edited image data, and (4) the manually generated image data; an inference generator that is equipped with a trained model of the trainable model obtained by a learning process by the learning processing device according to claim 2, or a trained model of the trainable model of the generator of the learning processing device according to claim 3 obtained by a learning process by the learning processing device according to claim 3, and that generates inference processing result image data from the map image data; a label input unit for inputting a label for the map image data; a generated data acquisition processing unit that acquires labeled image data that is a combination of the label inputted to the label input unit and the inference processing result image data generated by the inference generator; An image data generating device comprising:

5. an image input step of inputting image data; a standardization step of performing a standardization process on the image data input in the image input step to obtain standardized image data; a feature extraction image generating step of acquiring feature extraction image data by performing a feature extraction process on the standardized image data; a first editing processing step in which first editing data, which is data for editing the feature amount extracted image data, can be input, editing processing is performed on the feature amount extracted image data based on the first editing data, and image data after the editing processing is acquired as first edited image data; a second editing processing step in which second editing data, which is data for editing, can be input to the standardized image data, editing processing is performed on the standardized image data based on the second editing data, and the image data after the editing processing is acquired as second edited image data; a manually generated image input step for inputting manually generated image data which is data of a manually generated map image; a learning data generating step of acquiring learning data that is a combination of the standardized image data and any one of image data of (1) the feature amount extraction image data, (2) the first edited image data, (3) the second edited image data, and (4) the manually generated image data; A learning data generation method comprising:

6. A learning processing method for performing learning processing using learning data generated by the learning data generation method according to claim 5, comprising the steps of: a data input step of inputting the learning data generated by the learning data generating method according to claim 5 and extracting condition image data and actual image data from the learning data; a learning processing step of inputting the condition image data and the actual image data and performing a learning process of a trainable model; a loss evaluation step of evaluating an error between the data acquired by the learning processing step and training data; Equipped with Learning processing methods.

7. The learning process step includes: A generation step of inputting the condition image data and generating image data using a trainable model based on the input condition image data; an identification step of inputting the condition image data and either the image data generated by the generation step or the actual image data, and outputting a determination process result as to whether the input data is genuine data or fake data; Equipped with The learning processing method according to claim 6.

8. an image input step of inputting image data; a standardization step of performing a standardization process on the image data input in the image input step to obtain standardized image data; a feature extraction image generating step of acquiring feature extraction image data by performing a feature extraction process on the standardized image data; a first editing processing step in which first editing data, which is data for editing the feature amount extracted image data, can be input, editing processing is performed on the feature amount extracted image data based on the first editing data, and image data after the editing processing is acquired as first edited image data; a second editing processing step in which second editing data, which is data for editing, can be input to the standardized image data, editing processing is performed on the standardized image data based on the second editing data, and the image data after the editing processing is acquired as second edited image data; a manually generated image input step for inputting manually generated image data which is data of a manually generated map image; a map image generating step of acquiring, as map image data, one of image data of (1) the feature extraction image data, (2) the first edited image data, (3) the second edited image data, and (4) the manually generated image data; a generation step for inference of generating inference processing result image data from the map image data using a trained model of the learnable model obtained by a learning process according to the learning processing method of claim 6, or a trained model of the learnable model used in the generation step of the learning processing method of claim 7 obtained by a learning process according to the learning processing method of claim 7; a label input unit for inputting a label for the map image data; a generated data acquisition process step of acquiring labeled image data that is a combination of the label input to the label input unit and the inference processing result image data generated by the inference generation step; An image data generating method comprising:

9. A program for causing a computer to execute the learning data generating method according to claim 5.

10. A program for causing a computer to execute the learning processing method according to claim 6 or 7.

11. A program for causing a computer to execute the image data generating method according to claim 8.

Images