Generation device of data for learning, learning processing device, image data generation device, generation method of data for learning, learning processing method, generation method of image data and program

Abstract

To realize a generative system of learning data for medical diagnosis capable of collecting a large amount of highly accurate data for learning (medical image data (such as medical diagnostic image data)) required for obtaining a high-performance learned model through machine learning, while capable of performing an accurate annotation.SOLUTION: A generative system 1000 of learning data for medical diagnosis is capable of obtaining various pieces of image data for learning through a generation device 100 of data for learning. In the generative system 1000 of learning data for medical diagnosis, learning processing is performed by a learning processing device 200 by using the various pieces of image data for learning obtained through the generation device 100 of data for learning, to obtain a learned model. Thereafter, the learned model is installed to a medical image data generation device 300, and label inputting and inference processing are performed so that a large amount of medical image data with a label at high accuracy can be generated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for acquiring a large amount of highly accurate learning data to be used for learning to generate a medical AI model. [Background technology]

[0002] In recent years, a technology has been developed in which medical images (e.g., X-ray images) are input into a trained model obtained through machine learning, and diagnostic results are output from the trained model (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-32071 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional techniques described above, machine learning must be performed using medical diagnostic images, but in order to obtain a high-performance trained model through this machine learning, a large amount of highly accurate training data (medical diagnostic images) is required.

[0005] However, it is difficult to collect a large amount of highly accurate learning data (medical diagnostic images (medical data)) for the following reasons.

[0006] Medical data is individualized: each patient has their own unique symptoms and constitution, and there is a lack of sufficient data to cover this diversity.

[0007] Furthermore, medical data is extremely sensitive and requires strict security measures when it is handled, making it difficult to collect large amounts of medical data.

[0008] Additionally, medical data presents challenges in terms of data sharing, i.e., the complexity of sharing medical data while protecting patient privacy presents a barrier to data collection.

[0009] Furthermore, it is difficult to inexpensively annotate training data (medical diagnostic images (medical data)). In other words, annotating medical data (medical diagnostic images) requires highly specialized knowledge and expert judgment, which makes annotating training data (medical diagnostic images (medical data)) time-consuming and costly.

[0010] As such, it is difficult to collect large amounts of highly accurate training data (medical diagnostic images) and perform accurate annotation, which are necessary to obtain high-performance trained models through machine learning.

[0011] In view of the above problems, the present invention aims to provide a medical diagnostic training data generation system, a training data generation device, a training processing device, an image data generation device, a training data generation method, a training processing method, an image data generation method, and a program that can collect large amounts of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning and perform accurate annotation. [Means for solving the problem]

[0012] In order to solve the above problem, a first invention is a training data generation device comprising an image input unit, a standardization unit, a feature extraction image generation unit, a first editing processing unit, a second editing processing unit, a manually generated image input unit, and a training data generation unit.

[0013] The image input unit inputs image data.

[0014] The standardization unit performs standardization processing on the image data input to the image input unit, thereby obtaining standardized image data.

[0015] The feature extraction image generation unit performs feature extraction processing on the standardized image data to obtain feature extraction image data.

[0016] The first editing processing unit can input first editing data, which is data for editing the feature-extracted image data, and performs editing processing on the feature-extracted image data based on the first editing data, and obtains the image data after the editing processing as the first editing image data.

[0017] The second editing processing unit can input second editing data, which is data for editing, to the standardized image data, perform editing processing on the standardized image data based on the second editing data, and obtain the image data after the editing processing as the second editing image data. The manually generated image input unit is a functional unit for inputting manually generated image data, which is data for a manually generated map image.

[0018] The learning data generation unit acquires learning data that is a combination of standardized image data and one of the following image data: (1) feature-extracted image data, (2) first edited image data, (3) second edited image data, and (4) manually generated image data.

[0019] With the above configuration, this learning data generation device generates a variety of learning image data, i.e., (1) Learning data consisting of a combination of standardized image data (actual image data) and map image (condition image) data (image data with extracted features) automatically generated from the actual image data. (2) Learning data consisting of a combination of standardized image data (actual image data) and map image (condition image) data obtained by editing a map image automatically generated from the actual image data; (3) Learning data that combines standardized image data (actual image data) with map image (condition image) data obtained by performing editing processing based on the actual image data; (4) Learning data consisting of a combination of standardized image data (actual image data) and manually generated map image (condition image) data; can be acquired. Then, by using the various training image data acquired by this training data generation device to train a trainable model (for example, a conditional GAN model), highly accurate training processing can be performed. Then, by performing inference processing using the trained model acquired by the training processing, it is possible to acquire a large amount of highly accurate generated image data (for example, medical image data) even when only a small amount of actual image data can be acquired.

[0020] Note that "standardization processing" refers to a process in which, for example, an input image is subjected to extraction processing of a specified image area, padding processing of the specified image area, and image resizing processing to obtain standardized image data.

[0021] A second invention is the first invention, and is a learning processing device including a data input unit, a learning processing unit, and a loss evaluation unit.

[0022] The data input unit inputs the learning data generated by the learning data generating device of the first invention, and extracts condition image data and actual image data from the learning data.

[0023] The learning processing unit includes a trainable model, receives the condition image data and the actual image data, and performs a learning process for the trainable model.

[0024] The loss evaluation unit evaluates the error between the data output from the learning processing unit and the training data.

[0025] As a result, this learning processing device can perform a learning process for a trainable model using the learning data generated by the learning data generation device of the first invention. That is, even if only a small amount of actual image data can be acquired, the learning processing can be efficiently performed to obtain a trained model that can acquire a large amount of highly accurate generated image data (for example, medical image data).

[0026] A third invention is the second invention, in which the learning processing unit includes a generator and a classifier.

[0027] The generator has a trainable model that receives a condition image as input and generates image data based on the input condition image.

[0028] The classifier inputs the condition image and either the image data generated by the generator or the actual image data, and outputs the result of the determination process as to whether the input data is genuine data or fake data.

[0029] As a result, this learning processing device can perform learning processing by adopting, for example, a conditional GAN model as a trainable model.

[0030] In addition, the method of inputting the condition image and either the image data generated by the generator or the actual image data may be, for example, a method of inputting data that is a concatenation of the condition image and the image data generated by the generator, or a method of inputting data that is a concatenation of the condition image and the actual image data.

[0031] Furthermore, there may be a plurality of classifiers, and data of different scales (for example, downsampled data) may be input to the plurality of classifiers.

[0032] A fourth invention is an image data generation device comprising an image input unit, a standardization unit, a feature extraction image generation unit, a first editing processing unit, a second editing processing unit, a manually generated image input unit, a map image generation unit, an inference generator, a label input unit, and a generated data acquisition processing unit.

[0033] The image input unit inputs image data.

[0034] The standardization unit performs standardization processing on the image data input to the image input unit, thereby obtaining standardized image data.

[0035] The feature extraction image generation unit performs feature extraction processing on the standardized image data to obtain feature extraction image data.

[0036] The first editing processing unit can input first editing data, which is data for editing the feature-extracted image data, and performs editing processing on the feature-extracted image data based on the first editing image, and obtains the image data after the editing processing as the first editing image data.

[0037] The second editing processing unit can input second editing data, which is data for editing, to the standardized image data, perform editing processing on the standardized image data based on the second editing image, and obtain the image data after the editing processing as the second editing image data.

[0038] The manually generated image input unit inputs manually generated image data, which is data for a manually generated map image.

[0039] The map image generating unit acquires, as map image data, one of image data from (1) feature extraction image data, (2) first edited image data, (3) second edited image data, and (4) manually generated image data.

[0040] The inference generator is equipped with a trained model of a trainable model obtained by a learning process by the learning processing device described in the second invention, or a trained model of a trainable model of the generator of the learning processing device described in the third invention obtained by a learning process by the learning processing device described in the third invention, and generates inference processing result image data from map image data.

[0041] The label input unit inputs a label for the map image data.

[0042] The generated data acquisition processing unit acquires labeled image data that is a combination of the label input to the label input unit and the inference processing result image data generated by the inference generator.

[0043] With the above configuration, this image data generation device generates a variety of input image data (input data for inference processing) based on a small number of actual image data, i.e., (1) Map image (condition image) data (image data with extracted features) automatically generated from standardized actual image data, (2) Map image (condition image) data obtained by editing map images automatically generated from standardized real image data; (3) Map image (condition image) data obtained by editing based on standardized actual image data, (4) Manually generated map image (condition image) data, In this image data generation device, the various inference input data (image data) acquired as described above can be input to an inference generator equipped with a trainable model, thereby obtaining highly accurate generated image data. Furthermore, in the image data generation device, the user knows in advance what kind of data the image data (real image data) input to the second real image input unit 30 or the data input to the manually generated image input unit is (the user knows in advance that the real image data input to the image input unit or the data input to the manually generated image input unit is, for example, medical image data of a specific disease or medical image data of a health state), so data (information) for identifying the label corresponding to the map image output from the map image generation unit can be accurately input to the label input unit. Furthermore, the generated data acquisition processing unit of this image data generation device can obtain paired data (labeled data) consisting of the generated image data output from the inference generator and the label data output from the label input unit, so that accurately annotated data (labeled data) can be easily obtained in large quantities.

[0044] Therefore, this image data generation device can collect large amounts of highly accurate learning data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation. A fifth invention is a learning data generation method including an image input step, a standardization step, a feature extraction image generation step, a first editing processing step, a second editing processing step, a manual generated image input step, and a learning data generation step.

[0045] The image input step inputs image data.

[0046] In the standardization step, standardized image data is obtained by performing a standardization process on the image data input in the image input step.

[0047] The feature-extracted image generating step performs feature extraction processing on the standardized image data to obtain feature-extracted image data.

[0048] The first editing processing step can input first editing data, which is data for editing, to the feature-extracted image data, perform editing processing on the feature-extracted image data based on the first editing data, and obtain the image data after the editing processing as the first editing image data. The second editing processing step can input second editing data, which is data for editing, to the standardized image data, perform editing processing based on the second editing data on the standardized image data, and obtain the image data after the editing processing as the second editing image data.

[0049] The manually generated image input step inputs manually generated image data, which is data for a manually generated map image.

[0050] The learning data generation step acquires learning data that is a combination of standardized image data and one of the following image data: (1) feature-extracted image data, (2) first edited image data, (3) second edited image data, and (4) manually generated image data.

[0051] This makes it possible to realize a learning data generation method that has the same effects as the first aspect of the invention.

[0052] The sixth invention is a learning processing method for performing learning processing using the learning data generated by the learning data generation method of the fifth invention, and is a learning processing method comprising a data input step, a learning processing step, and a loss evaluation step. The data input step inputs the learning data generated by the learning data generating method according to the fifth aspect of the invention, and extracts condition image data and actual image data from the learning data. The learning process step inputs the condition image data and the actual image data and performs learning process of the trainable model. The loss evaluation step evaluates the error between the data acquired in the learning process step and the training data.

[0053] This makes it possible to realize a learning processing method that has the same effect as the second aspect of the invention.

[0054] A seventh aspect of the present invention is the sixth aspect of the present invention, wherein the learning process step includes a generating step and a identifying step.

[0055] The generation step inputs a condition image and generates image data using a trainable model based on the input condition image.

[0056] The identification step inputs a condition image and either the image data generated by the generation step or the actual image data, and outputs the result of the determination process as to whether the input data is genuine data or fake data.

[0057] This makes it possible to realize a learning processing method that has the same effect as the third aspect of the invention.

[0058] The eighth invention is an image data generation method including an image input step, a standardization step, a feature extraction image generation step, a first editing processing step, a second editing processing step, a manually generated image input step, a map image generation step, an inference generation step, a label input unit, and a generated data acquisition processing step.

[0059] The image input step inputs image data.

[0060] The standardization step performs standardization processing on the image data input in the image input step to obtain standardized image data.

[0061] The feature-extracted image generating step performs feature extraction processing on the standardized image data to obtain feature-extracted image data.

[0062] The first editing processing step can input first editing data, which is data for editing, to the feature-extracted image data, perform editing processing on the feature-extracted image data based on the first editing image, and obtain the image data after the editing processing as the first editing image data. The second editing processing step can input second editing data, which is data for editing, to the standardized image data, perform editing processing based on the second editing image on the standardized image data, and obtain the image data after the editing processing as the second editing image data.

[0063] The manually generated image input step inputs manually generated image data, which is data for a manually generated map image.

[0064] The map image generating step acquires, as map image data, one of image data (1) feature extraction image data, (2) first edited image data, (3) second edited image data, and (4) manually generated image data.

[0065] The inference generation step generates inference processing result image data from map image data using a trained model of a trainable model obtained by a learning process using the learning processing method described in the sixth invention, or a trained model of a trainable model used in the generation step of the learning processing method described in the seventh invention obtained by a learning process using the learning processing method described in the seventh invention.

[0066] The label input unit inputs a label for the map image data.

[0067] The generated data acquisition processing step acquires labeled image data that is a combination of the label input to the label input section and the inference processing result image data generated in the inference generation step.

[0068] This makes it possible to realize an image data generating method that has the same effect as the fourth aspect of the invention.

[0069] A ninth aspect of the present invention is a program for causing a computer to execute the learning data generation method of the fifth aspect of the present invention.

[0070] This makes it possible to realize a program for causing a computer to execute a learning data generation method that has the same effects as the fifth aspect of the invention.

[0071] A tenth aspect of the present invention is a program for causing a computer to execute the learning processing method of the sixth or seventh aspect of the present invention.

[0072] This makes it possible to realize a program for causing a computer to execute a learning processing method that has the same effects as the sixth or seventh aspect of the invention.

[0073] An eleventh aspect of the present invention is a program for causing a computer to execute the image data generating method of the eighth aspect of the present invention.

[0074] This makes it possible to realize a program for causing a computer to execute an image data generating method that has the same effect as the eighth aspect of the invention. [Effects of the Invention]

[0075] According to the present invention, it is possible to realize a medical diagnostic training data generation system, a training data generation device, a training processing device, an image data generation device, a training data generation method, a training processing method, an image data generation method, and a program that can collect large amounts of high-precision training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning and perform accurate annotation. [Brief explanation of the drawings]

[0076] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a medical diagnosis learning data generation system 1000 according to a first embodiment. [Figure 2] 1 is a diagram showing a schematic configuration of a training data generation device 100 of a medical diagnosis training data generation system 1000 according to a first embodiment. [Figure 3] FIG. 1 is a diagram showing a schematic configuration of a learning processing device 200 of a medical diagnosis learning data generation system 1000 according to a first embodiment. [Figure 4] 1 is a diagram showing a schematic configuration of a medical image data generation device 300 of a medical diagnosis training data generation system 1000 according to a first embodiment. [Figure 5] FIG. 3 is a diagram for explaining the processing (standardization processing) of the standardization unit 12. [Figure 6] FIG. 3 is a diagram for explaining the processing (standardization processing) of the standardization unit 12. [Figure 7] 3 is a diagram for explaining the processing (feature extraction processing) of the feature extraction image generating unit 13. FIG. [Figure 8] 4 is a diagram for explaining the processing of the first edit processing unit 14 (first edit processing). FIG. [Figure 9] FIG. 4 is a diagram for explaining the processing (second editing processing) of the second editing processing unit 15. [Figure 10] A diagram showing the CPU bus configuration. DETAILED DESCRIPTION OF THE INVENTION

[0077] [First embodiment] The first embodiment will be described below with reference to the drawings.

[0078] <1.1: Configuration of the medical diagnosis learning data generation system> FIG. 1 is a diagram showing a schematic configuration of a medical diagnosis learning data generation system 1000 according to the first embodiment.

[0079] FIG. 2 is a diagram showing a schematic configuration of the training data generation device 100 of the medical diagnosis training data generation system 1000 according to the first embodiment.

[0080] FIG. 3 is a diagram showing a schematic configuration of the learning processing device 200 of the medical diagnosis learning data generation system 1000 according to the first embodiment.

[0081] FIG. 4 is a diagram showing a schematic configuration of the medical image data generation device 300 of the medical diagnosis training data generation system 1000 according to the first embodiment.

[0082] As shown in FIG. 1, the medical diagnosis training data generation system 1000 includes a training data generation device 100, a first data storage unit DB1, a learning processing device 200, a medical image data generation device 300, and a second data storage unit DB2.

[0083] (1.1.1: Learning data generation device 100) As shown in FIG. 2, the training data generation device 100 includes an actual image input unit 11, a standardization unit 12, a feature extraction image generation unit 13, a first editing processing unit 14, a second editing processing unit 15, a manually generated image input unit 16, and a training data generation unit 17.

[0084] The real image input unit 11 is a functional unit (e.g., a data input interface) for inputting actual medical image data (real medical image data). The real image input unit 11 outputs data including the input medical image data to the standardization unit 12 as data Din_real.

[0085] The standardization unit 12 inputs the data Din_real output from the real image input unit 11, acquires the medical image data (data of the real medical image) included in the data Din_real, and performs a standardization process on the medical image data (data of the real medical image).The standardization unit 12 then outputs data including the data after the standardization process (image data) as data D1_real to the feature extraction image generation unit 13, the second edit processing unit 15, and the learning data generation unit 17.

[0086] The feature extraction image generation unit 13 inputs the data D1_real output from the standardization unit 12, performs feature extraction image generation processing on the standardized image data included in the data D1_real, and generates (acquires) a feature extraction image through this processing.The feature extraction image generation unit 13 then outputs data including the feature extraction image generated (acquired) through the feature extraction image generation processing as data D1_map_auto to the first edit processing unit 14 and the learning data generation unit 17.

[0087] The first edit processing unit 14 receives as input the data D1_map_auto output from the feature extraction image generation unit 13 and data D11_stylus indicating the editing content (the data D11_stylus is, for example, data input from an external source). The first edit processing unit 14 performs an edit process on the feature extraction image included in the data D1_map_auto based on the data D11_stylus indicating the editing content, and obtains image data after the edit process. The first edit processing unit 14 then outputs data including the image data after the edit process as data D11_map_edited to the learning data generation unit 17.

[0088] The second edit processing unit 15 receives as input the data D1_real output from the standardization unit 12 and data D12_stylus indicating the edit content (the data D12_stylus is, for example, data input from an external source). The second edit processing unit 15 performs an edit process on the post-standardization image (image data) included in the data D1_real based on the data D12_stylus indicating the edit content, and obtains the post-edit image data. The second edit processing unit 15 then outputs data including the post-edit image data to the learning data generation unit 17 as data D12_map_edited.

[0089] Manually generated image input unit 16 is a functional unit (e.g., a data input interface) for inputting manually generated image data. Manually generated image input unit 16 outputs data including the input image data to learning data generation unit 17 as data D1_map_manual.

[0090] The learning data generation unit 17 receives as input the data D1_real output from the standardization unit 12, the data D1_map_auto output from the feature extraction image generation unit 13, the data D11_map_edited output from the first edit processing unit 14, the data D12_map_edited output from the second edit processing unit 15, and the data D1_map_manual output from the manually generated image input unit 16. The learning data generation unit 17 also receives as input a mode signal Mode. The mode signal Mode is generated by a control unit (not shown) that controls the functional units of the learning data generation device 100, and is output from the control unit to the learning data generation unit 17.

[0091] In accordance with the mode signal Mode, the training data generation unit 17 acquires training data (training data in which standardized real image data and map image data (condition image data) are paired) by pairing (1) data D1_real output from the standardization unit 12 with one of (2A) data D1_map_auto output from the feature extraction image generation unit 13, (2B) data D11_map_edited output from the first edit processing unit 14, (2C) data D12_map_edited output from the second edit processing unit 15, and (2D) data D1_map_manual output from the manually generated image input unit 16. Then, the training data generation unit 17 outputs the acquired training data as data Dtrain_pair (for example, to the first data storage unit DB1).

[0092] (1.1.2: First Data Storage Unit DB1) The first data storage unit DB1 is a functional unit that can read data from a predetermined area (memory area) of the first data storage unit DB1 in accordance with an external data read command and / or can write data to a predetermined area (memory area) of the first data storage unit DB1 in accordance with an external data write command. The first data storage unit DB1 is realized by, for example, a data storage device, a database, or a memory device.

[0093] In accordance with a data write command from the device 100 for generating training data, the first data storage unit DB1 writes the data Dtrain_pair output from the training data generation unit 17 of the device 100 for generating training data into a predetermined area (memory area) of the first data storage unit DB1.

[0094] In addition, in accordance with a data read command from the learning processing device 200, the first data storage unit DB1 reads data stored in a specified area (memory area) of the first data storage unit DB1 from that area, and outputs the read data to the learning processing device 200.

[0095] (1.1.3: Learning processing device 200) 3, the learning processing device 200 includes a data input unit 21, a generator 22, a first combination unit 23A, a second combination unit 23B, a selector SEL1, a downsampling unit 24, a first classifier 25A, a second classifier 25B, and a loss evaluation unit 26. The learning processing device 200 is a device for executing a learning process using a generative adversarial network model (GAN model (GAN: Generative Adversarial Network)) having a conditional input, a so-called conditional GAN model (learnable model).

[0096] The data input unit 21 outputs a data read command to the first data storage unit DB1, reads out learning data (learning data in which standardized actual image data and map image data (condition image data) are paired) stored in a predetermined area (memory area) of the first data storage unit DB1, and inputs the read learning data as data Din_pair (learning data Din_pair). The data input unit 21 extracts map image data (condition image data) from the learning data Din_pair, and outputs the extracted map image data (condition image data) as data Din_condition to the generator 22, the first combination unit 23A, and the second combination unit 23B.

[0097] Furthermore, the data input unit 21 extracts standardized real image data from the learning data Din_pair, and outputs the extracted standardized real image data to the second combining unit 23B as data Din_real.

[0098] The generator 22 is a generator of a conditional GAN model, and is a trainable model (the parameters of the generator are θ (G) The generator 22 inputs the map image data (condition image data) Din_condition output from the data input unit 21. The generator 22 inputs the map image data (condition image data) Din_condition to the trainable model of the generator 22, executes processing using the trainable model, and outputs the data acquired by the processing to the first combination unit 23A as data Do_G.

[0099] The first combining unit 23A inputs the data Do_G output from the generator 22 and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Do_G and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sG.

[0100] The second combining unit 23B inputs the data Din_real and data Din_condition output from the data input unit 21, performs a process of concatenating the data Din_real and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sx.

[0101] The selector SEL1 is a two-input, one-output selector that selects either the data Do_G output from the first combining unit 23A or the data Do_sx output from the second combining unit 23B in accordance with a selection signal sel1 from a control unit (not shown) that controls the functional units of the learning processing device 200, and outputs the selected data as data Do0 to the downsampling unit 24 and the second discriminator 25B.

[0102] The downsampling unit 24 inputs the data Do0 output from the selector SEL1, performs a downsampling process on the data Do0, obtains the downsampled data, and outputs the obtained data as data Do1 to the first discriminator 25A.

[0103] The first discriminator 25A is a discriminator of a conditional GAN model, and is a trainable model (the parameters of the first discriminator are set as θ (D1) and includes T (T: natural number) layers (layers that extract features). The first classifier 25A receives the data Do1 output from the downsampling unit 24. The first classifier 25A receives the data Do1 output from the downsampling unit 24 as input to the trainable model of the first classifier 25A, executes processing using the trainable model (processing to determine whether the input data Do1 is genuine or fake), and outputs data including determination result data obtained by the processing and output data (feature data) of the T (T: natural number) layers (layers that extract features) to the loss evaluation unit 26 as data Do_D1.

[0104] The second discriminator 25B is a discriminator of a conditional GAN model, and is a trainable model (the parameters of the second discriminator are set as θ (D2) The second classifier 25B receives the data Do0 output from the selector SEL1 as input. The second classifier 25B executes processing using the trainable model (processing for determining whether the input data Do0 is genuine or fake), and outputs data including the determination result data acquired by the processing and the output data (feature data) of the T layers (layers for extracting features) as data Do_D2 to the loss evaluation unit 26.

[0105] The loss evaluation unit 26 receives the determination result data Do_D1 output from the first classifier 25A, the determination result data Do_D2 output from the second classifier 25B, and the teacher data D_correct, and obtains an error (loss obtained by a loss function) based on the determination result data Do_D1 and the teacher data D_correct, and an error (loss obtained by a loss function) based on the determination result data Do_D2 and the teacher data D_correct. Then, the loss evaluation unit 26 performs parameter update processing (learning processing) of the first classifier 25A, the second classifier 25B, and the trainable model of the generator 22 based on the obtained error (loss) (for example, performs learning processing using an error backpropagation method). The teacher label (corresponding to the teacher data D_correct) for the judgment data as to whether it is genuine or fake, contained in the output data Dout_D1 and Dout_D2 of the trainable model of the first classifier 25A and the second classifier 25B, is “1” indicating that it is genuine data when data including real data (including data after downsampling) is input to the first classifier 25A and the second classifier 25B, and is “0” indicating that it is fake data when data including fake data (output data of the generator 22) (including data after downsampling) is input to the first classifier 25A and the second classifier 25B.

[0106] In addition, the loss evaluation unit 26 inputs the data Din_condition (the probability distribution of this data is set to "s") output from the data input unit 21, the data Do_G (the probability distribution of this data is set to "G(s)") output from the generator 22, and the data Din_real (the probability distribution of this data is set to "x") output from the data input unit 21, and evaluates the loss of the conditional GAN model using a loss function based on the input data, and performs a determination process to determine whether the learning process has converged.

[0107] (1.1.4: Medical image data generation device 300) As shown in Figure 4, the medical image data generation device 300 includes a second actual image input unit 30, a second standardization unit 31, a second feature extraction image generation unit 32, a third editing processing unit 33, a fourth editing processing unit 34, a second manually generated image input unit 35, a map image generation unit 36, a label input unit 37, a generator 38, and a generated data acquisition processing unit 39.

[0108] The second real image input unit 30 is a functional unit (e.g., a data input interface) for inputting actual medical image data (real medical image data). The second real image input unit 30 outputs data including the input medical image data to the second standardization unit 31 as data Din2_real.

[0109] The second standardization unit 31 inputs the data Din2_real output from the second actual image input unit 30, acquires the medical image data (data of the actual medical image) included in the data Din2_real, and performs standardization processing on the medical image data (data of the actual medical image).The second standardization unit 31 then outputs data including the data after the standardization processing (image data) as data D2_real to the second feature extraction image generation unit 32 and the third edit processing unit 33.

[0110] The second feature extraction image generation unit 32 receives the data D2_real output from the second standardization unit 31, performs feature extraction image generation processing on the standardized image data included in the data D2_real, and generates (acquires) a feature extraction image through this processing.The second feature extraction image generation unit 32 then outputs data including the feature extraction image generated (acquired) through the feature extraction image generation processing to the third edit processing unit 33 as data D2_map_auto.

[0111] The third edit processing unit 33 receives data D2_map_auto output from the second feature extraction image generation unit 32 and data D21_stylus indicating the editing content (the data D21_stylus is, for example, data input from an external source). The third edit processing unit 33 performs an edit process on the feature extraction image included in the data D21_map_auto based on the data D21_stylus indicating the editing content, and obtains image data after the edit process. The third edit processing unit 33 then outputs data including the image data after the edit process as data D21_map_edited to the map image generation unit 36.

[0112] The fourth edit processing unit 34 receives as input the data D2_real output from the second standardization unit 31 and data D22_stylus indicating the editing content (the data D22_stylus is, for example, data input from an external source). The fourth edit processing unit 34 performs an edit process on the post-standardization image (image data) included in the data D2_real based on the data D22_stylus indicating the editing content, and obtains the post-edit image data. The fourth edit processing unit 34 then outputs data including the post-edit image data to the map image generation unit 36 as data D22_map_edited.

[0113] Second manually generated image input unit 35 is a functional unit (e.g., a data input interface) for inputting manually generated image data. Second manually generated image input unit 35 outputs data including the input image data to map image generation unit 36 as data D2_map_manual.

[0114] The map image generation unit 36 receives as input the data D2_map_auto output from the second feature extraction image generation unit 32, the data D21_map_edited output from the third edit processing unit 33, the data D22_map_edited output from the fourth edit processing unit 34, and the data D2_map_manual output from the second manually generated image input unit 35. The map image generation unit 36 also receives as input a mode signal Mode. The mode signal Mode is generated by a control unit (not shown) that controls the functional units of the medical image data generation device 300, and is output from the control unit to the map image generation unit 36.

[0115] In accordance with the mode signal Mode, the map image generation unit 36 selects one of the following data: (1) data D2_map_auto output from the second feature extraction image generation unit 32, (2) data D21_map_edited output from the third edit processing unit 33, (3) data D22_map_edited output from the fourth edit processing unit 34, and (4) data D2_map_manual output from the second manually generated image input unit 35, and outputs the selected data to the generator 38 as data Do_map.

[0116] The label input unit 37 is a functional unit (for example, a data input interface) for inputting data (information) for identifying a label corresponding to the map image output from the map image generation unit 36. The label input unit 37 outputs the input data (label data) to the generated data acquisition processing unit 39 as data Lbl.

[0117] The generator 38 is a generator of a conditional GAN model, and has the same function and configuration as the generator 22 of the learning processing device 200. The generator 38 includes a trained model (a model of a generator that has completed the learning process by the learning processing device 200 (optimal parameters are θ opt (G) The generator 38 is equipped with a model in which the following conditions are set: the generator 38 inputs map image data (condition image data) Do_map output from the map image generation unit 36; the generator 38 inputs the map image data (condition image data) Do_map into the trained model of the generator 38, executes processing using the trained model, and outputs the data acquired by the processing to the generated data acquisition processing unit 39 as data D_gene_out.

[0118] The generated data acquisition processing unit 39 receives the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37. The generated data acquisition processing unit 39 performs a process (generated data acquisition process) to combine the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37 into a set of data (paired data (labeled data)), and outputs the data acquired by this process to the second data storage unit DB2 as data Dout (= {D_gene_out, Lbl}).

[0119] (1.1.5: Second data storage unit DB2) The second data storage unit DB2 is a functional unit that can read data from a predetermined area (memory area) of the second data storage unit DB2 in accordance with an external data read command and / or write data to a predetermined area (memory area) of the second data storage unit DB2 in accordance with an external data write command. The second data storage unit DB2 is realized by, for example, a data storage device, a database, or a memory device.

[0120] In accordance with a data write command from the medical image data generation device 300, the second data storage unit DB2 writes the data Dout output from the generated data acquisition processing unit 39 of the medical image data generation device 300 into a specified area (memory area) of the second data storage unit DB2.

[0121] <1.2: Operation of the medical diagnosis learning data generation system> The operation of the medical diagnosis learning data generation system 1000 configured as above will be described with reference to the drawings.

[0122] In the following, the operation of the medical diagnosis training data generation system 1000 will be explained separately for (1) training data generation processing, (2) training processing, and (3) medical image data generation processing.

[0123] 5 and 6 are diagrams for explaining the processing (standardization processing) of the standardization unit 12. FIG.

[0124] FIG. 7 is a diagram for explaining the processing (feature extraction processing) of the feature extraction image generating unit 13. In FIG.

[0125] FIG. 8 is a diagram for explaining the processing (first editing processing) of the first editing processing unit 14. In FIG.

[0126] FIG. 9 is a diagram for explaining the processing (second editing processing) of the second edit processing unit 15. In FIG.

[0127] (1.2.1: Learning data generation process) First, the learning data generation process will be described.

[0128] Actual medical image data (real medical image data) is input to the real image input unit 11 (for example, a user inputs real medical image data from outside to the real image input unit 11). The real image input unit 11 outputs data including the input medical image data to the standardization unit 12 as data Din_real.

[0129] The standardization unit 12 inputs the data Din_real output from the real image input unit 11, acquires the medical image data (data of the real medical image) included in the data Din_real, and performs standardization processing on the medical image data (data of the real medical image). For example, the standardization unit 12 performs standardization processing on the medical image data (data of the real medical image) by performing (1) a process of cutting out (extracting) a predetermined region (image region) of the image (for example, a process of cutting out (extracting) the left and right regions (or the top and bottom regions) of the image), (2) a process of padding the predetermined region of the image (padding processing), (3) a process of changing the size of the image, and so on, and acquires a standardized image (for example, an image of a predetermined size) through the standardization processing.

[0130] For example, when the image shown in Figure 5 is input to the standardization unit 12 as medical image data (data of an actual medical image), the standardization unit 12 performs (1) a process of cutting out the left and right regions of the image, (2) a padding process (a process of padding the left and right image regions with zeros after the cutout process), and (3) a resizing process, thereby obtaining an image (the image shown on the right side of Figure 5) cut out from the left and right image regions of the input medical image data (data of an actual medical image) as a standard image (for example, an image with an image size of x1 x y1).

[0131] Furthermore, when the image shown in Figure 6 is input to the standardization unit 12 as medical image data (data of an actual medical image), the standardization unit 12 can obtain an image (the image shown on the right side of Figure 6) cut out from a specified image area of the input medical image data (data of an actual medical image) as a standard image (for example, an image with an image size of x1 x y1) by performing (1) padding processing (processing to pad the left and right image areas with zeros after the cutout processing) and (2) resizing processing.

[0132] The standardization unit 12 outputs data including the standardized data (image data) to the feature extraction image generation unit 13, the second edit processing unit 15, and the learning data generation unit 17 as data D1_real.

[0133] The feature extraction image generation unit 13 inputs the data D1_real output from the standardization unit 12, performs feature extraction image generation processing on the image data after standardization processing included in the data D1_real, and generates (obtains) a feature extraction image through this processing.

[0134] For example, when the data (standardized image) D1_real after standardization processing shown in Fig. 7 is input to the feature extraction image generation unit 13, the feature extraction image generation unit 13 executes feature extraction image generation processing (e.g., edge extraction processing) on the input standard image, and obtains an image from which features have been extracted (e.g., an edge extraction image) (e.g., the image shown on the right side of Fig. 7). Note that in Fig. 7, the feature extraction image (e.g., the edge extraction image) is displayed in a negative / positive inverted state.

[0135] Then, the feature extraction image generation unit 13 outputs data including the feature extraction image generated (acquired) by the feature extraction image generation process to the first edit processing unit 14 and the learning data generation unit 17 as data D1_map_auto.

[0136] The first edit processing unit 14 receives as input the data D1_map_auto output from the feature extraction image generation unit 13 and data D11_stylus (the data D11_stylus is, for example, data input from an external source) that indicates the editing content. The first edit processing unit 14 performs editing processing on the feature extraction image included in the data D1_map_auto based on the data D11_stylus that indicates the editing content, and obtains image data after the editing processing.

[0137] For example, when the feature extraction image data (e.g., edge extraction image) D1_map_auto shown in FIG. 8 is input to the first edit processing unit 14, the first edit processing unit 14 displays the feature extraction image data (e.g., edge extraction image) D1_map_auto on, for example, a touch panel display device (not shown), and performs an edit process on the feature extraction image data (e.g., edge extraction image) D1_map_auto by, for example, having the user draw or erase in a predetermined image area on the displayed image with a touch pen. The data of the edit process is then input to the first edit processing unit 14 as data D11_stylus. The first edit processing unit 14 performs an edit process on the feature extraction image data (e.g., edge extraction image) D1_map_auto based on the data D11_stylus instructing the editing content, and obtains data of the image after the edit process (e.g., the image shown on the right side of FIG. 8). Note that in FIG. 8, the feature extraction image (e.g., edge extraction image) and the image after the edit process are displayed in a negative / positive inverted state.

[0138] Then, the first edit processing unit 14 outputs data including the edited image data to the learning data generation unit 17 as data D11_map_edited.

[0139] The second edit processing unit 15 receives data D1_real of the standardized image output from the standardization unit 12 and data D12_stylus that indicates the editing content (the data D12_stylus is, for example, data input from an external source). The second edit processing unit 15 performs an edit process on the standardized image included in the standardized image data D1_real based on the data D12_stylus that indicates the editing content, and obtains the image data after the edit process.

[0140] For example, when the standardized image data D1_real shown in FIG. 9 is input to the second edit processing unit 15, the second edit processing unit 15 displays the standardized image data D1_real on, for example, a touch panel display device (not shown), and performs an edit process on the standardized image data D1_real (for example, a process of obtaining image data created by drawing by tracing the edge portions of the standardized image) by having the user draw (for example, by tracing the edge portions of the standardized image) or erase, for example, in a predetermined image area on the displayed image with a touch pen. The data of the edit process is then input to the second edit processing unit 15 as data D12_stylus. The second edit processing unit 15 performs an edit process on the standardized image data D1_real based on the data D12_stylus instructing the editing content, and obtains data of the image after the edit process (for example, the image shown on the right side of FIG. 9). Note that the image on the right side of FIG. 9 is displayed with negative / positive inversion.

[0141] Then, the second edit processing unit 15 outputs data including the edited image data to the learning data generation unit 17 as data D12_map_edited.

[0142] The manually generated image input unit 16 is a functional unit (e.g., a data input interface) for inputting manually generated image data, and outputs data including the input image data as data D1_map_manual to the learning data generation unit 17. For example, the manually generated image input unit 16 has a touch panel display device (not shown), and on the touch panel display device, a user draws and creates an image similar to the feature extraction image data (e.g., edge extraction image) D1_map_auto using, for example, a touch pen. Then, the manually generated image input unit 16 outputs data including the input image data (an image manually created by the user (an image similar to the feature extraction image data (e.g., edge extraction image) D1_map_auto)) to the learning data generation unit 17 as data D1_map_manual.

[0143] The learning data generation unit 17 receives as input the data D1_real output from the standardization unit 12, the data D1_map_auto output from the feature extraction image generation unit 13, the data D11_map_edited output from the first edit processing unit 14, the data D12_map_edited output from the second edit processing unit 15, and the data D1_map_manual output from the manually generated image input unit 16. The learning data generation unit 17 also receives as input a mode signal Mode from a control unit (not shown).

[0144] In accordance with the mode signal Mode, the learning data generation unit 17 pairs (1) the data D1_real output from the standardization unit 12 with one of (2A) the data D1_map_auto output from the feature extraction image generation unit 13, (2B) the data D11_map_edited output from the first editing processing unit 14, (2C) the data D12_map_edited output from the second editing processing unit 15, and (2D) the data D1_map_manual output from the manually generated image input unit 16, thereby obtaining learning data (learning data in which standardized real image data and map image data (condition image data) are paired).

[0145] For example, the learning data generating unit 17 performs the following processing in accordance with the signal value of the mode signal Mode. (1) When the signal value of the mode signal Mode is “1”, the training data generation unit 17 acquires data (={D1_real, D1_map_auto}) that is a pair of data D1_real output from the standardization unit 12 and data D1_map_auto output from the feature extraction image generation unit 13 as training data Dtrain_pair. (2) When the signal value of the mode signal Mode is “2”, the training data generation unit 17 acquires the data (={D1_real, D11_map_edited}) that is a pair of the data D1_real output from the standardization unit 12 and the data D11_map_edited output from the first editing processing unit 14 as training data Dtrain_pair. (3) When the signal value of the mode signal Mode is “3”, the training data generation unit 17 acquires the data (={D1_real, D12_map_edited}) that is a pair of the data D1_real output from the standardization unit 12 and the data D12_map_edited output from the second editing processing unit 15 as training data Dtrain_pair. (4) When the signal value of the mode signal Mode is “4”, the learning data generation unit 17 acquires data (={D1_real, D1_map_manual}) that is a pair of data D1_real output from the standardization unit 12 and data D1_map_manual output from the manually generated image input unit 16 as learning data Dtrain_pair.

[0146] Then, the learning data generating unit 17 outputs the acquired learning data Dtrain_pair to the first data storage unit DB1.

[0147] Then, in accordance with a data write command from the learning data generation device 100, the first data storage unit DB1 writes the data Dtrain_pair output from the learning data generation unit 17 of the learning data generation device 100 into a predetermined area (memory area) of the first data storage unit DB1.

[0148] (1.2.2: Learning process) Next, the learning process will be described.

[0149] The data input unit 21 of the learning processing device 200 outputs a data read command to the first data storage unit DB1, reads out learning data (learning data in which standardized actual image data and map image data (condition image data) are paired) stored in a predetermined area (memory area) of the first data storage unit DB1, and inputs the read learning data as data Din_pair (learning data Din_pair). The data input unit 21 extracts map image data (condition image data) from the learning data Din_pair and outputs the extracted map image data (condition image data) as data Din_condition to the generator 22, the first combination unit 23A, and the second combination unit 23B.

[0150] Furthermore, the data input unit 21 extracts standardized real image data from the learning data Din_pair, and outputs the extracted standardized real image data to the second combining unit 23B as data Din_real.

[0151] Generator 22, which is the generator of the conditional GAN model (the generator parameters are θ (G) The data input unit 23A inputs map image data (condition image data) Din_condition output from the data input unit 21. The generator 22 inputs the map image data (condition image data) Din_condition to the trainable model of the generator 22, executes processing using the trainable model, and outputs the data acquired by the processing to the first combination unit 23A as data Do_G.

[0152] The first combining unit 23A inputs the data Do_G output from the generator 22 and the data Din_condition output from the data input unit 21, performs a process of concatenating the data Do_G and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sG.

[0153] The second combining unit 23B inputs the data Din_real and data Din_condition output from the data input unit 21, performs a process of concatenating the data Din_real and the data Din_condition, and outputs the concatenated data to the selector SEL1 as data Do_sx.

[0154] The selector SEL1 selects either the data Do_sG output from the first combining unit 23A or the data Ds_sx output from the second combining unit 23B in accordance with a selection signal sel1 from a control unit (not shown) that controls the functional units of the learning processing device 200. When the signal value of the selection signal sel is "0," the selector SEL1 selects the data Do_G output from the generator 22 (data input to terminal 0) and outputs the data Do_G as data Do0 to the downsampling unit 24 and the second discriminator 25B. On the other hand, when the signal value of the selection signal sel is "1," the selector SEL1 selects the data Do_sx output from the second combining unit 23B (data input to terminal 1) and outputs the data Do_sx as data Do0 to the downsampling unit 24 and the second discriminator 25B.

[0155] The downsampling unit 24 inputs the data Do0 output from the selector SEL1, performs a downsampling process on the data Do0, obtains the downsampled data, and outputs the obtained data as data Do1 to the first discriminator 25A.

[0156] The first discriminator 25A (the discriminator parameter is θ (D1)The first classifier 25A inputs the data Do1 output from the downsampling unit 24 to its trainable model, executes processing by the trainable model (determines whether the input data Do1 is genuine or fake), and outputs data including the determination result data acquired by the processing and the output data (feature data) of the T layers (layers that extract features) as data Do_D1 to the loss evaluation unit 26. The trainable model of the first classifier 25A outputs “1” when it determines that the input data Do1 is genuine, and outputs “0” when it determines that the input data D2 is fake.

[0157] The second discriminator 25B (the discriminator parameters are θ (D2) The second classifier 25B inputs the data Do0 output from the selector SEL1 to its trainable model, executes processing by the trainable model (processing to determine whether the input data Do0 is genuine or fake), and outputs data including the determination result data acquired by the processing and the output data (feature data) of the T layers (layers that extract features) as data Do_D2 to the loss evaluation unit 26. The trainable model of the second classifier 25B outputs "1" if it determines that the input data Do1 is genuine, and outputs "0" if it determines that the input data D2 is fake.

[0158] The teacher label (corresponding to the teacher data D_correct) for the judgment data as to whether it is genuine or fake, contained in the output data Dout_D1 and Dout_D2 of the trainable model of the first classifier 25A and the second classifier 25B, is “1” indicating that it is genuine data when data including real data (including data after downsampling) is input to the first classifier 25A and the second classifier 25B, and is “0” indicating that it is fake data when data including fake data (output data of the generator 22) (including data after downsampling) is input to the first classifier 25A and the second classifier 25B.

[0159] The loss evaluation unit 26 obtains the determination result data Do_D1 output from the first classifier 25A, the determination result data Do_D2 output from the second classifier 25B, an error (loss obtained by a loss function) based on the determination result data Do_D1 and the teacher data D_correct, and an error (loss obtained by a loss function) based on the determination result data Do_D2 and the teacher data D_correct. Then, the loss evaluation unit 26 performs parameter update processing (learning processing) of the first classifier 25A, the second classifier 25B, and the trainable model of the generator 22 based on the obtained error (loss) (for example, performs learning processing by error backpropagation).

[0160] Specifically, the learning processing device 200 performs the learning process by performing processing corresponding to the following formula:

number

number

[0161] Furthermore, the feature matching loss function LFM(G,D) for the generator G (generator 22) and the k-th classifier (the first classifier 25A and the second classifier 25B) is k ) is as follows:

number

[0162] Then, (1) the loss function L for the generator G (the generator 22) and the classifier D (the first classifier 25A and the second classifier 25B) obtained as described above is calculated. GAN (G,D k ), and (2) a feature matching loss function L for the generator G (generator 22) and the k-th classifier (the first classifier 25A and the second classifier 25B). FM (G,D k ), the learning processing device 200 performs processing equivalent to (Equation 1), thereby obtaining the parameter θ (G) update process of the first classifier 25A, the parameter θ (D1) and the parameter θ (D2) The parameter θ of the generator 22 is updated. (G) The optimal parameter θ opt (G) , the parameter θ of the first classifier 25A (D1) The optimal parameter θ opt (D1) , and the parameter θ of the second classifier 25B (D2) The optimal parameter θ opt (D2) In the learning processing device 200, the parameter θ (G) The optimal parameter θ opt (G) , the parameter θ of the first classifier 25A (D1) The optimal parameter θ opt (D1) , and the parameter θ of the second classifier 25B (D2) The optimal parameter θopt (D2) Once this is acquired, the learning process is terminated.

[0163] (1.2.3: Medical image data generation processing) Next, the medical image data generation process will be described.

[0164] The generator 38 of the medical image data generating device 300 is provided with the optimal parameters θ of the generator acquired by the learning processing device 200 through the learning process. opt (G) It is equipped with a model (pre-trained model) with the following settings.

[0165] Actual medical image data (real medical image data) is input to the second real image input unit 30 (for example, a user inputs real medical image data from outside to the second real image input unit 30). The second real image input unit 30 outputs data including the input medical image data to the second standardization unit 31 as data Din2_real.

[0166] The second standardization unit 31 inputs the data Din2_real output from the second actual image input unit 30, acquires the medical image data (data of the actual medical image) included in the data Din2_real, and performs standardization processing on the medical image data (data of the actual medical image). For example, the second standardization unit 31 performs standardization processing on the medical image data (data of the actual medical image) by performing (1) a process of cutting out (extracting) a predetermined region (image region) of the image (e.g., a process of cutting out (extracting) the left and right regions (or the top and bottom regions) of the image), (2) a process of padding the predetermined region of the image (padding processing), and (3) a process of resizing the image, and thereby acquires a standardized image (e.g., an image of a predetermined size) through the standardization processing (performing processing similar to that of the standardization unit 12).

[0167] Then, the second standardization unit 31 outputs data including the standardized data (image data) as data D2_real to the second feature amount extraction image generation unit 32 and the fourth edit processing unit .

[0168] The second feature extraction image generation unit 32 inputs the data D2_real output from the second standardization unit 31, performs a feature extraction image generation process on the image data after the standardization process included in the data D2_real, and generates (obtains) a feature extraction image through this process (performs the same process as the feature extraction image generation unit 13).

[0169] Then, the second feature amount extraction image generating unit 32 outputs data including the feature amount extraction image generated (acquired) by the feature amount extraction image generating process to the third edit processing unit 33 and the map image generating unit 36 as data D2_map_auto.

[0170] The third edit processing unit 33 receives as input the data D2_map_auto output from the second feature extraction image generation unit 32 and data D21_stylus that indicates the editing content (the data D21_stylus is, for example, data input from an external source). The third edit processing unit 33 performs editing processing on the feature extraction image included in the data D2_map_auto based on the data D21_stylus that indicates the editing content, and obtains image data after the editing processing (performing processing similar to that of the first edit processing unit 14).

[0171] Then, the third edit processing unit 33 outputs data including the image data after the edit processing to the map image generating unit 36 as data D21_map_edited.

[0172] The fourth edit processing unit 34 receives the standardized image data D2_real output from the second standardization unit 31 and data D22_stylus that indicates the editing content (the data D22_stylus is, for example, data input from an external source). The fourth edit processing unit 34 performs an edit process on the standardized image included in the standardized image data D2_real based on the data D22_stylus that indicates the editing content, and obtains the image data after the edit process (performing the same process as the second edit processing unit 15).

[0173] Then, the fourth edit processing unit 34 outputs data including the edited image data to the map image generating unit 36 as data D22_map_edited.

[0174] The second manually generated image input unit 35 is a functional unit (e.g., a data input interface) for inputting manually generated image data, and outputs data including the input image data to the map image generation unit 36 as data D2_map_manual. For example, the second manually generated image input unit 35 has a touch panel display device (not shown), and on the touch panel display device, a user draws and creates an image similar to the feature extraction image data (e.g., an edge extraction image) D2_map_auto, for example, using a touch pen. Then, the second manually generated image input unit 35 outputs data including the input image data (an image manually created by the user (an image similar to the feature extraction image data (e.g., an edge extraction image) D2_map_auto)) to the map image generation unit 36 as data D2_map_manual.

[0175] Map image generation unit 36 receives as input data D2_map_auto output from second feature amount extraction image generation unit 32, data D21_map_edited output from third edit processing unit 33, data D22_map_edited output from fourth edit processing unit 34, and data D2_map_manual output from second manually generated image input unit 35. Map image generation unit 36 also receives as input a mode signal Mode from a control unit (not shown).

[0176] In accordance with the mode signal Mode, the map image generation unit 36 acquires one of the following data as map image data Do_map to be output to the generator 38: (1) data D2_map_auto output from the second feature extraction image generation unit 32, (2) data D21_map_edited output from the third editing processing unit 33, (3) data D22_map_edited output from the fourth editing processing unit 34, and (4) data D2_map_manual output from the second manually generated image input unit 35.

[0177] For example, the learning data generating unit 17 performs the following processing in accordance with the signal value of the mode signal Mode. (1) When the signal value of the mode signal Mode is "1", the map image generating unit 36 acquires the data D2_map_auto output from the second feature amount extraction image generating unit 32 as map image data Do_map. (2) When the signal value of the mode signal Mode is "2," the map image generating unit 36 acquires the data D21_map_edited output from the third edit processing unit 33 as the map image data Do_map. (3) When the signal value of the mode signal Mode is "3," the map image generating unit 36 acquires the data D22_map_edited output from the fourth edit processing unit 34 as the map image data Do_map. (4) When the signal value of the mode signal Mode is "4," the map image generating unit 36 acquires the data D2_map_manual output from the second manually generated image input unit 35 as the map image data Do_map.

[0178] Then, map image generating unit 36 outputs the map image data Do_map obtained as described above to generator 38.

[0179] The generator 38 is provided with a trained model (a model of a generator that has completed training processing by the training processing device 200 (optimal parameters are θ opt (G) The generator 38 is equipped with a model in which the following conditions are set: the generator 38 inputs map image data (condition image data) Do_map output from the map image generation unit 36; the generator 38 inputs the map image data (condition image data) Do_map into the trained model of the generator 38, executes processing using the trained model, and outputs the data acquired by the processing to the generated data acquisition processing unit 39 as data D_gene_out.

[0180] Data (information) for identifying a label corresponding to the map image output from map image generation unit 36 is input to label input unit 37. Then, label input unit 37 outputs the input data (label data) as data Lbl to generated data acquisition processing unit 39. Note that the user knows in advance what kind of data the actual image data input to second actual image input unit 30 or the data input to second manually generated image input unit 35 is (the user knows in advance that the actual image data input to second actual image input unit 30 or the data input to second manually generated image input unit 35 is, for example, medical image data of a person suffering from a specific disease or medical image data of a healthy state), and therefore can accurately input data (information) for identifying a label corresponding to the map image output from map image generation unit 36 to label input unit 37.

[0181] The generated data acquisition processing unit 39 receives the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37. The generated data acquisition processing unit 39 performs a process (generated data acquisition process) to combine the data D_gene_out output from the generator 38 and the data Lbl output from the label input unit 37 into a set of data (paired data (labeled data)), and outputs the data acquired by this process to the second data storage unit DB2 as data Dout (= {D_gene_out, Lbl}).

[0182] By repeatedly executing the above process in the medical image data generation device 300, a large amount of highly accurate labeled medical image data can be generated.

[0183] <Summary> As described above, in the medical diagnosis training data generation system 1000, the training data generation device 100 generates various training image data, i.e., (1) Learning data (learning data in automatic generation mode) that combines standardized actual image data with map image (condition image) data (image data from which features have been extracted) automatically generated from the actual image data. (2) Learning data (learning data in semi-automatic mode (map image editing mode)) that combines standardized actual image data with map image (condition image) data obtained by editing a map image automatically generated from the actual image data. (3) Learning data (learning data in semi-automatic mode (real image editing mode)) that combines standardized real image data with map image (condition image) data obtained by editing the real image data. (4) Training data (training data in manual mode) consisting of standardized actual image data and manually generated map image (condition image) data; can be acquired. In the medical diagnosis training data generation system 1000, the training processing device 200 can train a trainable model (e.g., a conditional GAN model) using various training image data acquired by the training data generation device 100. That is, in the training process of the medical diagnosis training data generation system 1000, the training processing device 200 can train a trainable model (e.g., a conditional GAN model) using actual image data, which can only be acquired in small amounts, and map image data acquired based on the actual image data. Therefore, even if only a small amount of actual image data can be acquired, a large amount of training data for training a trainable model can be acquired. As a result, the training processing device 200 can train a trainable model (e.g., a conditional GAN model) with high accuracy.

[0184] In the medical diagnosis learning data generation system 1000, the medical image data generation device 300 generates various input image data (input data for inference processing) based on a small number of actual image data, i.e., (1) Map image (condition image) data (image data with extracted features) automatically generated from standardized real image data (input data for inference using automatic generation mode), (2) Map image (condition image) data obtained by editing a map image automatically generated from standardized real image data (input data for inference in semi-automatic mode (map image editing mode)), (3) Map image (condition image) data obtained by editing based on standardized real image data (input data for inference in semi-automatic mode (real image editing mode)), (4) Manually generated map image (condition image) data (input data for inference in manual mode), can be acquired. In the medical diagnosis training data generation system 1000, the various inference input data (image data) acquired as described above are input to a generator 38 equipped with a trained model, thereby enabling highly accurate generated image data D_gene_out to be acquired. Furthermore, in the medical diagnosis training data generation system 1000, the user knows in advance what kind of data the actual image data input to the second actual image input unit 30 of the medical image data generation device 300 or the data input to the second manually generated image input unit 35 is (the user knows in advance that the actual image data input to the second actual image input unit 30 or the data input to the second manually generated image input unit 35 is, for example, medical image data of a subject suffering from a specific disease or medical image data of a healthy state), and therefore data (information) for identifying a label corresponding to the map image output from the map image generation unit 36 can be accurately input to the label input unit 37. The generated data acquisition processing unit 39 of the medical image data generation device 300 can acquire data (paired data (labeled data)) Dout, which is a combination of data D_gene_out output from the generator 38 and data Lbl output from the label input unit 37, so that accurately annotated data (labeled data) can be easily acquired in large quantities.

[0185] As described above, the medical diagnostic training data generation system 1000 can collect a large amount of highly accurate training data (medical image data (e.g., medical diagnostic image data)) required to obtain a high-performance trained model through machine learning, and can perform accurate annotation.

[0186] [Other embodiments] In the above embodiment, a case where a conditional GAN model is adopted in the learning processing device 200 of the medical diagnosis training data generation system 1000 has been described, but this is not limited thereto, and other generation models that allow condition input (for example, a diffusion model that allows condition input) may also be adopted. That is, the learning processing device 200 may perform a learning process using other generation models that allow condition input (for example, a diffusion model that allows condition input), and the trained model acquired by the learning process may be loaded into the medical image data generation device 300, which may then perform inference processing (medical image data generation processing).

[0187] Furthermore, in the medical diagnostic training data generation system 1000 described in the above embodiment, each block (each functional unit) may be individually implemented as a single chip using a semiconductor device such as an LSI, or may be integrated into a single chip to include some or all of the blocks. Furthermore, each block (each functional unit) of the medical diagnostic training data generation system 1000, training data generation device 100, training processing device 200, and medical image data generation device 300 described in the above embodiment may be realized by multiple semiconductor devices such as LSIs.

[0188] Although we refer to it as an LSI here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the level of integration.

[0189] Furthermore, the method of integration is not limited to LSI, but may be realized by dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow the connections and settings of circuit cells within LSI to be reconfigured, may also be used.

[0190] Furthermore, some or all of the processing of each functional block in each of the above embodiments may be realized by a program. And, some or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. Furthermore, the programs for performing each processing are stored in a storage device such as a hard disk or ROM, and are executed in the ROM or by being read into the RAM.

[0191] Furthermore, each process in the above-described embodiments may be realized by hardware, or by software (including cases where it is realized together with an OS (operating system), middleware, or a predetermined library). Furthermore, it may be realized by a combination of software and hardware.

[0192] When each functional unit of the above embodiment is realized by software, for example, each functional unit may be realized by software processing using the hardware configuration shown in FIG. 10 (for example, a hardware configuration in which a CPU (which may be a GPU), a ROM, a RAM, an input unit, an output unit, a communication unit, a memory unit (for example, a memory unit realized by an HDD, an SSD, etc.), an external media drive, etc. are connected via a bus).

[0193] Furthermore, when each functional unit of the above embodiment is realized by software, the software may be realized using a single computer having the hardware configuration shown in Figure 10, or may be realized by distributed processing using multiple computers.

[0194] Furthermore, the execution order of the processing method in the above embodiment is not necessarily limited to the description of the above embodiment, and the execution order can be changed within the scope of the gist of the invention. Furthermore, in the processing method in the above embodiment, some steps may be executed in parallel with other steps within the scope of the gist of the invention.

[0195] The scope of the present invention includes a computer program for causing a computer to execute the above-described method, and a computer-readable recording medium having the program recorded thereon. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories.

[0196] The computer program is not limited to one recorded on the recording medium, but may be one transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, or the like.

[0197] The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention. [Explanation of symbols]

[0198] 1000 Medical diagnosis learning data generation system 100 Learning data generation device 200 Learning processing device 300 Medical image data generation device 11 Real image input unit 12 Standardization Department 13 Feature extraction image generation unit 14 First editing processing section 15 Second Editing Processing Section 16 Manually generated image input section 17 Learning data generation unit 21 Data Entry Section 22 Generator 25A 1st discriminator 25B 1st discriminator 26 Loss Assessment Department 30 Second actual image input unit 31 2nd Standardization Department 32 Second feature extraction image generation unit 33 Third Editing Processing Unit 34 4th Editing Processing Unit 35 Second manually generated image input unit 36 Map image generation unit 37 Label input section 38 Generator (inference generator) 39 Generated data acquisition processing unit

Claims

1. an image input unit for inputting image data; a standardization unit that performs a standardization process on the image data input to the image input unit to obtain standardized image data; a feature extraction image generation unit that performs feature extraction processing on the standardized image data to obtain feature extraction image data; a first edit processing unit that can input first edit data, which is data for editing the feature amount extracted image data, performs edit processing on the feature amount extracted image data based on the first edit data, and acquires the image data after the edit processing as first edited image data; a second editing processing unit that can input second editing data, which is data for editing the standardized image data, performs editing processing on the standardized image data based on the second editing data, and acquires the image data after the editing processing as second edited image data; a manually generated image input unit for inputting manually generated image data, which is data of a manually generated map image; a learning data generation unit that acquires learning data that is a combination of the standardized image data and any one of image data of (1) the feature amount extracted image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data; A learning data generation device comprising:

2. A learning processing device for performing learning processing using the learning data generated by the learning data generation device according to claim 1, a data input unit that inputs the learning data generated by the learning data generation device according to claim 1 and extracts condition image data and actual image data from the learning data; a learning processing unit including a trainable model, receiving the condition image data and the actual image data, and performing a learning process for the trainable model; a loss evaluation unit that evaluates an error between the data output from the learning processing unit and training data; Equipped with Learning processing device.

3. The learning processing unit A generator including a trainable model that receives the condition image and generates image data based on the input condition image; a classifier that receives the condition image and either the image data generated by the generator or the real image data, and outputs a determination process result as to whether the input data is real data or fake data; Equipped with The learning processing device according to claim 2 .

4. an image input unit for inputting image data; a standardization unit that performs a standardization process on the image data input to the image input unit to obtain standardized image data; a feature extraction image generation unit that performs feature extraction processing on the standardized image data to obtain feature extraction image data; a first edit processing unit that can input first edit data, which is data for editing the feature amount extracted image data, performs edit processing on the feature amount extracted image data based on the first edited image, and acquires the image data after the edit processing as first edited image data; a second editing processing unit that can input second editing data, which is data for editing the standardized image data, performs editing processing on the standardized image data based on the second editing image, and acquires the image data after the editing processing as second editing image data; a manually generated image input unit for inputting manually generated image data, which is data of a manually generated map image; a map image generation unit that acquires, as map image data, image data of any one of (1) the feature extraction image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data; an inference generator that is equipped with a trained model of the trainable model obtained by a learning process by the learning processing device according to claim 2, or a trained model of the trainable model of the generator of the learning processing device according to claim 3 obtained by a learning process by the learning processing device according to claim 3, and that generates inference processing result image data from the map image data; a label input unit for inputting a label for the map image data; a generated data acquisition processing unit that acquires labeled image data that is a combination of the label inputted to the label input unit and the inference processing result image data generated by the inference generator; An image data generating device comprising:

5. an image input step of inputting image data; a standardization step of performing a standardization process on the image data input in the image input step to obtain standardized image data; a feature-extracted image generating step of acquiring feature-extracted image data by performing feature extraction processing on the standardized image data; a first editing processing step in which first editing data, which is data for editing the feature amount extracted image data, can be input, editing processing is performed on the feature amount extracted image data based on the first editing data, and image data after the editing processing is acquired as first edited image data; a second editing processing step in which second editing data, which is data for editing the standardized image data, can be input to the standardized image data, editing processing is performed on the standardized image data based on the second editing data, and the image data after the editing processing is acquired as second edited image data; a manually generated image input step for inputting manually generated image data, which is data of a manually generated map image; a learning data generating step of acquiring learning data that is a combination of the standardized image data and any one of image data of (1) the feature amount extracted image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data; A learning data generation method comprising:

6. A learning processing method for performing learning processing using learning data generated by the learning data generation method according to claim 5, comprising: a data input step of inputting the learning data generated by the learning data generation method according to claim 5 and extracting condition image data and actual image data from the learning data; a learning processing step of inputting the condition image data and the actual image data and performing learning processing of a trainable model; a loss evaluation step of evaluating an error between the data acquired by the learning processing step and training data; Equipped with Learning processing methods.

7. The learning processing step includes: a generation step of inputting the condition image and generating image data using a trainable model based on the input condition image; an identification step of inputting the condition image and either the image data generated by the generation step or the actual image data, and outputting a determination process result as to whether the input data is genuine data or fake data; Equipped with The learning processing method according to claim 6 .

8. an image input step of inputting image data; a standardization step of performing a standardization process on the image data input in the image input step to obtain standardized image data; a feature-extracted image generating step of acquiring feature-extracted image data by performing feature extraction processing on the standardized image data; a first editing processing step in which first editing data, which is data for editing the feature amount extracted image data, can be input, editing processing is performed on the feature amount extracted image data based on the first editing image, and image data after the editing processing is acquired as first edited image data; a second editing processing step in which second editing data, which is data for editing the standardized image data, can be input to the standardized image data, editing processing is performed on the standardized image data based on the second editing image, and the image data after the editing processing is acquired as second editing image data; a manually generated image input step for inputting manually generated image data, which is data of a manually generated map image; a map image generating step of acquiring, as map image data, image data of any one of (1) the feature amount extracted image data, (2) the first edited image data, (3) the first edited image data, and (4) the manually generated image data; a generation step for inference, which generates inference processing result image data from the map image data using a trained model of the trainable model obtained by a learning process by the learning processing method according to claim 6, or a trained model of the trainable model used in the generation step of the learning processing method according to claim 7 obtained by a learning process by the learning processing method according to claim 7; a label input unit for inputting a label for the map image data; a generated data acquisition processing step of acquiring labeled image data that is a combination of the label input to the label input unit and the inference processing result image data generated by the inference generation step; An image data generation method comprising:

9. A program for causing a computer to execute the learning data generation method according to claim 5.

10. A program for causing a computer to execute the learning processing method according to claim 6 or 7.

11. A program for causing a computer to execute the image data generating method according to claim 8.

Images