Data generation device, data generation method and program

Abstract

To provide a technology that enables easier generation of learning data related to images. [Solution] A data generation device is provided that has a display control unit that displays an image and the boundaries of multiple partial images generated by dividing the image into multiple parts in an overlapping manner on a screen, an input unit that accepts input of labels to be assigned to each of the multiple partial images, and a generation unit that generates learning data for training a learning model by associating each of the multiple partial images with the labels assigned to each of the multiple partial images.

Description

[Technical field]

[0001] The present invention relates to a data generation device, a data generation method, and a program. [Background technology]

[0002] Currently, there are known systems that use artificial intelligence to analyze pathological images and perform pathological diagnosis. For example, Patent Document 1 describes a method of training a machine learning algorithm using a plurality of annotated digital pathological images, and using a discrimination model generated by the training to identify whether an abnormal image pattern corresponds to a pathological abnormality. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-525151 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to train a learning model, it is necessary to prepare a large amount of pathological images containing labels indicating the presence or absence of abnormalities as learning data. At this time, since a doctor needs to judge whether or not an abnormality is included in a pathological image, there is a problem that a large amount of learning data is generated, which places a heavy burden on the doctor. Note that the same problem is not limited to pathological images, but may occur with any type of image.

[0005] Therefore, an object of the present invention is to provide a technique that enables learning data related to images to be generated more easily. [Means for solving the problem]

[0006] A data generation device according to one embodiment of the present invention has a display control unit that displays on a screen an image and the boundaries of multiple partial images generated by dividing the image into multiple parts in an overlapping manner, an input unit that accepts input of labels to be assigned to each of the multiple partial images, and a generation unit that generates learning data for training a learning model by matching each of the multiple partial images with the labels assigned to each of the multiple partial images. Effect of the Invention

[0007] According to the present invention, it is possible to provide a technique that enables learning data related to an image to be generated more easily. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of an image processing system that determines whether or not an abnormality exists in a pathology image. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a data generating device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional block configuration of a data generating device. [Figure 4] 11 is a flowchart illustrating an example of a processing procedure performed by a data generating device. [Diagram 5] FIG. 13 is a diagram showing an example of a screen for labeling. [Figure 6] FIG. 13 is a diagram for explaining a process of dividing a pathological image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An embodiment of the present invention will be described with reference to the attached drawings. In each drawing, the same reference numerals are attached to the same or similar configurations. In this embodiment, a case where learning data related to a pathological image is generated from an input pathological image is described as an example, but is not limited to this. This embodiment can be applied to a case where learning data is generated from various images. For example, it can be applied to two-dimensional image data related to medical applications such as ultrasound images, CT images, and endoscopic images. It can also be applied to images other than medical applications.

[0010] <System configuration> FIG. 1 is a diagram showing an example of an image processing system 1 that judges whether or not an abnormality exists in a pathological image. The data generating device 10 is a device that generates learning data (teacher data) for training a learning model provided in a diagnostic device 20, using a pathological image input to the data generating device 10. The diagnostic device 20 is a device that judges whether or not an abnormality exists in a pathological image, using a trained model trained using the learning data generated by the data generating device 10. The data generating device 10 and the diagnostic device 20 can communicate with each other via a wireless or wired communication network N.

[0011] The data generating device 10 displays on a screen a pathological image input to the data generating device 10 and the boundaries of a plurality of partial images generated by dividing the pathological image into a plurality of parts, with the boundaries overlapping. The partial images may be called tiles. The size of the partial images (the number of pixels in the vertical and horizontal directions) is the same as the size of learning data that can be input to the learning model. The data generating device 10 also accepts input of a label related to pathology to be assigned to each partial image displayed on the screen from a pathologist or the like (hereinafter referred to as a "user") who uses the data generating device 10. In the following description, the label is described on the premise that there are two types of labels indicating whether or not a pathological abnormality exists in the partial image, but the present embodiment is not limited to this. The present embodiment can also be applied to a case where three or more types of labels are assigned.

[0012] The data generating device 10 may generate learning data from one pathological image, or may generate learning data from multiple pathological images. When generating learning data from multiple pathological images, the data generating device 10 repeats a process of dividing the pathological image into multiple partial images and accepting input of labels for each pathological image. When the user has completed labeling of all pathological images, the data generating device 10 generates learning data by associating image data of each partial image with the label assigned to each partial image. The generated learning data is sent to the diagnostic device 20.

[0013] The diagnostic device 20 trains a learning model using the learning data sent from the data generating device 10. The learning model is, for example, a neural network that can be given a predetermined ability through learning. The diagnostic device 20 inputs a pathological image of a diagnosis target to the trained model generated by learning, and determines whether or not an abnormality exists in the pathological image based on the output result from the trained model.

[0014] 1, the data generating device 10 is illustrated as one information processing device, but the present embodiment is not limited to this. For example, the data generating device 10 may be configured with one or more physical servers, or may be configured with a virtual server operating on a hypervisor, or may be configured with a cloud server.

[0015] <Hardware configuration> 2 is a diagram showing an example of a hardware configuration of the data generating device 10. The data generating device 10 includes a processor 11 such as a CPU (Central Processing Unit) or a GPU (Graphical processing unit), a storage device 12 such as a memory, an HDD (Hard Disk Drive) and / or an SSD (Solid State Drive), a communication IF (Interface) 13 for wired or wireless communication, an input device 14 for accepting an input operation, and an output device 15 for outputting information. The input device 14 is, for example, a keyboard, a touch panel, a mouse, and / or a microphone. The output device 15 is, for example, a display, a touch panel, and / or a speaker.

[0016] <Function block configuration> FIG. 3 is a diagram showing an example of a functional block configuration of the data generating device 10. The data generating device 10 includes a storage unit 100, a display control unit 101, an input unit 102, a generating unit 103, and an image processing unit 104. The storage unit 100 can be realized by using a storage device 12 included in the data generating device 10. The display control unit 101, the input unit 102, the generating unit 103, and the image processing unit 104 can be realized by the processor 11 of the data generating device 10 executing a program stored in the storage device 12. The program can be stored in a storage medium. The storage medium storing the program may be a non-transitory computer readable storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a storage medium such as a USB memory or a CD-ROM.

[0017] The memory unit 100 stores a pathology image DB (image DB) that stores one or more pathological images (images) used to generate learning data, a label DB that stores labels assigned to partial images by a user, and a learning data DB that stores the generated learning data.

[0018] The display control unit 101 causes an output device 15 such as a display, or other information processing devices communicating with the data generating device 10 to display various screens according to the present embodiment. The display control unit 101 also causes a screen in which the boundary line of a partial image is superimposed on the pathological image (image) to be displayed.

[0019] The input unit 102 accepts various inputs from the user. The input unit 102 also accepts input of a label to be assigned to each of the multiple partial images from the user via the input device 14. The input unit 102 stores the label assigned to each partial image in the label DB. For example, the input unit 102 may store in the label DB a partial image ID that uniquely identifies each partial image and a label assigned to each partial image in association with each other. The label may be input by, for example, selecting a partial image to which a label is to be input from among the partial images displayed on the screen, and accepting designation of a label to be assigned to the partial image.

[0020] In addition, when generating learning data from multiple pathological images (multiple images), the input unit 102 may be configured to accept input of information regarding the type of pathology corresponding to the multiple pathological images (a type of pathology common to the multiple pathological images, an image type corresponding to the multiple images).

[0021] The generating unit 103 generates learning data for training a learning model by associating each of the multiple partial images with a label assigned to each of the multiple partial images by the input unit 102. For example, the generating unit 103 acquires a partial image ID and a label from a label DB, and extracts image data of a partial image corresponding to the partial image ID from a pathology image (image) stored in a pathology image DB (image DB). Next, the generating unit 103 generates learning data by combining the image data of the extracted partial image with the label corresponding to the partial image ID.

[0022] Image processing unit 104 changes the colors of the multiple pathological images (multiple images) according to a color change method determined according to the type of pathology (image type) input to input unit 102. The change method may include a method of changing the RGB values ​​of each pixel in the multiple pathological images (multiple images) to a standardized value, and a method of randomly changing the RGB values ​​of each pixel in the multiple pathological images (multiple images).

[0023] <Processing Procedure> FIG. 4 is a flowchart showing an example of a processing procedure performed by the data generating device 10. First, the input unit 102 accepts an input of a pathological image used to generate learning data from a user, and stores the input pathological image in a pathological image DB (S10). A pathological image is usually stained to make it easier to determine whether it is abnormal or not. Examples of staining methods include hematoxylin-eosin staining, PAS staining, May-Giemsa staining, Alcian blue staining, Papanicolaou staining, Azan staining, Elastica van Gieson staining, and Elastica Masson staining. When a user inputs multiple pathological images, the user inputs pathological images with the same pathological type. In the following description, it is assumed that a user inputs multiple pathological images.

[0024] Next, the input unit 102 accepts input of information on the type of pathology for the multiple pathology images input in the processing procedure of step S10 from the user (S11). The type of pathology that can be input may be, for example, either "tumor" or "hypermutation." If each of the multiple pathology images is an image of a tumor, the user inputs "tumor" as the type of pathology, and if each of the multiple pathology images is an image of hypermutation, the user inputs "hypermutation" as the type of pathology.

[0025] Next, the image processing unit 104 changes the colors of the multiple pathological images according to the color change method according to the type of pathology input in the processing procedure of step S11 (S12). For example, when the type of pathology is "tumor", the color change method may be to change the RGB values ​​of each pixel in the multiple pathological images to standardized values. First, the image processing unit 104 calculates the average value and standard deviation of R (Red) values, the average value and standard deviation of G (Green) values, and the average value and standard deviation of B (Blue) values ​​for all pixels in the multiple pathological images. Next, the image processing unit 104 standardizes the R values ​​of each pixel in the multiple pathological images using the average value and standard deviation of the R values. Specifically, the standardized R value can be calculated by using the formula "standardized R value = (R value - average value of R values) ÷ standard deviation of R values". The same calculation is performed for the G value and the B value to calculate the standardized G value and the standardized B value. This makes it possible to uniform the color tone of the multiple pathological images.

[0026] Also, when the type of pathology is "hypermutation", the color change method may be a method of randomly shifting the RGB values ​​of each pixel in a plurality of pathological images for each pathological image. First, the image processing unit 104 randomly determines the number of times to shift the RGB values ​​for each pathological image, and shifts the R value, G value, and B value by the determined value. For example, when the image processing unit 104 determines the number of times to shift the RGB values ​​for the first pathological image to be "5", it adds 5 to the R value, G value, and B value of each pixel included in the pathological image. Next, when the image processing unit 104 determines the number of times to shift the RGB values ​​for the second pathological image to be "-2", it adds -2 to the R value, G value, and B value of each pixel included in the pathological image. The image processing unit 104 repeats the same process for all pathological images. This makes it possible to vary the colors of the plurality of pathological images.

[0027] Next, the display control unit 101 selects one pathological image from the input pathological images, and displays a screen in which the boundary line of the partial image is superimposed on the selected pathological image (S13). Next, the input unit 102 accepts input of a label to be assigned to each partial image (S14). When the input pathological images are images of tumors, the label assigned by the user to each partial image is either a label indicating that the partial image is an image of a tumor, or a label indicating that the partial image is not an image of a tumor. When the input pathological images are images of hypermutations, the label assigned by the user to each partial image is either a label indicating that the partial image is an image of hypermutations, or a label indicating that the partial image is not an image of hypermutations.

[0028] The input unit 102 stores the received label in the label DB in association with the partial image ID. If the input of the label for each partial image is not completed (S15-No), the data generating device 10 repeats the processing procedure of step S14. If the input of the label for each partial image is completed (S15-Yes), the generating unit 103 generates learning data by associating the assigned label with the image data of each partial image for which labeling has been completed (S16). The determination of whether the input of the label is completed may be performed based on whether the input unit 102 detects that a predetermined button (e.g., a button for starting the generation of learning data, etc.) is pressed on the screen. Alternatively, the determination may be performed based on whether labels are assigned to a predetermined number of partial images. In addition, when labels are assigned to a predetermined number of partial images and the ratio of the assigned labels (the ratio for each type of label) is within a predetermined range (e.g., the ratio of the label indicating the image of a tumor to the label indicating the image of a non-tumor is approximately 1:4, etc.), the assignment of the labels may be automatically determined to be completed.

[0029] The generating unit 103 may store the generated learning data in the learning data DB by dividing the data into folders for each label. For example, image data of partial images labeled as having a tumor and image data of partial images labeled as not having a tumor may be stored in separate folders.

[0030] In the above-described processing procedure, partial images in which tissue is not present at a predetermined ratio or more (partial images that satisfy a predetermined condition) may be excluded from the learning data. For example, in the processing procedure of step S14, the input unit 102 may not accept input of a label for a partial image in which a specific color portion (e.g., a white portion) is present at a predetermined ratio or more among the plurality of partial images in each pathological image. In addition, in the processing procedure of step S16, the generation unit 103 may not include, among the plurality of partial images, partial images for which the input unit 102 did not accept input of a label, in the learning data to be generated. This makes it possible to prevent the generation of learning data that is inappropriate for learning a learning model.

[0031] FIG. 5 is a diagram showing an example of a screen for labeling. FIG. 5 shows an example in which the pathological image is an image including tumor cells. Selection menu M10 is a menu for designating the type of pathological image input by the user. Selection menu M11 is a menu for designating the pathological image to be displayed on the screen when multiple pathological images are input. The example in FIG. 5 shows that 40 pathological images related to tumor cells have been input, and the third pathological image is currently being displayed.

[0032] In the display area W10 of FIG. 5, a grid-like boundary line is superimposed on an image in which a part of a pathological image is enlarged. In the display area W10, one area surrounded by a boundary line corresponds to one partial image. In the overview display area W11, an image of the entire pathological image and a display frame V11 indicating an area of ​​the pathological image that is enlarged and displayed in the display area W10 are displayed. The position or size of the display frame V11 may be arbitrarily changed by, for example, a mouse operation. When the input unit 102 accepts a change in the position or size of the display frame V11, the display control unit 101 changes the pathological image to be enlarged and displayed in the display area W10 according to the position and size of the changed display frame V11.

[0033] The size of the partial image (the number of pixels in the vertical and horizontal directions) is predetermined according to the learning model to be learned, and may not be changed by the user. In the example of Fig. 5, the size of the partial image is displayed as 300 pixels in the vertical direction and 300 pixels in the horizontal direction.

[0034] Also displayed in the display area W10 is a display frame T10 that indicates the position of a partial image for which a label is to be input by the user. The position of the display frame T10 can be changed as desired by the user selecting a partial image for which the user wishes to input a label. By displaying the display frame T10, the user can recognize the position of the partial image to which the user is about to assign a label.

[0035] For partial images for which label input has been completed, information indicating the input label is displayed superimposed on the partial image. In the example of Fig. 5, the letter "T" is displayed in the upper left corner of a partial image to which a label indicating that the partial image is an image of a tumor cell has been added, and the letter "N" is displayed in the upper left corner of a partial image to which a label indicating that the partial image is not an image of a tumor cell has been added.

[0036] The display control unit 101 may display both information indicating the ratio of each label, which is the target value of the label to be assigned to each of the plurality of partial images, and information indicating the actual value of the ratio of each label assigned to the plurality of partial images, on the labeling screen. For example, assume that there exists a condition in which the recognition accuracy of the learning model is high when the ratio of the image of a tumor cell to an image of a non-tumor cell is set to 1:4 as learning data. In the example of FIG. 5, a criterion "the target value of the label ratio of T and N is T:N=1:4" is displayed as a target value in the memo display area A10 that displays the criterion for labeling. This indicates that it is desirable to assign the labels so that the ratio between the number of partial images labeled with T and the number of partial images labeled with N is 1:4. In addition, the display area N10 displays the ratio of the labels already assigned as information indicating the actual value of the ratio of each label assigned to the plurality of partial images. 5, it is desirable to set the ratio between the number of partial images labeled with T and the number of partial images labeled with N to 1:4, but currently, it is shown that the ratio between the number of partial images labeled with T and the number of partial images labeled with N is 1:2. Therefore, the user can recognize that it is necessary to search for and label partial images that are not images of tumor cells until the ratio displayed in the display area N10 becomes 1:4.

[0037] The display control unit 101 may also display both the target value of the number of partial images to be labeled and the information indicating the actual value of the number of partial images to which labels have been added on the labeling screen. For example, assume that there exists a condition that the recognition accuracy of the learning model is high when 2,000 images are learned as learning data. In the example of FIG. 5, a standard "The target number of tiles to be labeled is 2,000" is displayed as a target value in the memo display area A10. This indicates that it is desirable for the user who adds the labels to add labels to 2,000 partial images. In addition, the display area N11 displays the actual value of the number of partial images to which labels have been added. This allows the user to recognize that it is necessary to add labels to another 1,000 partial images.

[0038] The partial images may be images generated by dividing a pathological image from a reference point (e.g., the upper left pixel) at a predetermined interval in the right direction and downward direction (e.g., 300 pixel intervals in the right direction and 300 pixel intervals in the downward direction, etc.). Without being limited thereto, the partial images may be images generated by dividing a point shifted a predetermined distance from a reference point in the pathological image at a predetermined interval in the left-right direction and upward-downward direction (e.g., 300 pixel intervals in the right-left direction and 300 pixel intervals in the upward-downward direction, etc.). For example, the partial images may be images generated by dividing a pathological image at a predetermined interval in the right direction and downward direction based on a point (pixel) shifted a predetermined distance in the right direction and / or downward direction from a reference point (upper left image) in the pathological image. The offset value designation area M12 is an area for designating how many pixels the reference point in the pathological image is to be shifted to the right. The offset value designation area M13 is an area for designating how many pixels the reference point in the pathological image is to be shifted downward. When the designation of the predetermined distance is accepted by the input unit 102, the display control unit 101 divides the partial image based on a point shifted by the designated predetermined distance from the top left point of the pathological image, and displays the divided partial image on the screen.

[0039] A specific example will be described with reference to FIG. 6. In the case where partial images are generated by dividing the pathological image at a predetermined interval in the right direction and the down direction based on the top left of the pathological image, partial image P1 indicates the partial image located at the top left of the pathological image, and partial image P6 indicates the partial image located to the right of partial image P1. In this state, it is assumed that 50 pixels and 0 pixels are inputted into offset value designation areas M12 and M13 in FIG. 5, respectively. In this case, partial image P2 is generated based on a point shifted 50 pixels to the right from the top left of the pathological image. Similarly, it is assumed that 100 pixels and 0 pixels are inputted into offset value designation areas M12 and M13 in FIG. 5, respectively. In this case, partial image P3 is generated based on a point shifted 100 pixels to the right from the top left of the pathological image.

[0040] <Summary> According to the embodiment described above, the data generating device 10 divides an image into a plurality of partial images and displays them, and receives a label input from the user for each partial image. This makes it possible to generate learning data related to images more easily.

[0041] In addition, the data generating device 10 changes pixel values ​​related to the colors of the multiple pathological images according to a pixel value changing method determined according to the image type corresponding to the multiple images. This makes it possible to change the colors of the multiple images to more appropriate colors suitable for learning the learning model according to the image type, and to generate a learning model with higher recognition accuracy.

[0042] Furthermore, the data generating device 10 displays a target value and an actual value for the ratio of each type of label to be assigned to each partial image. This allows the user to assign a label to each partial image so that the actual value approaches the target value, thereby making it possible to efficiently generate learning data for generating a learning model with higher recognition accuracy.

[0043] Furthermore, the data generating device 10 enables an image to be divided into partial images based on any point (pixel) on the image as a reference point. This allows a user to repeat the labeling process by shifting the reference point used for dividing the image into partial images after labeling, thereby generating a large amount of learning data even when the number of input images is small.

[0044] The above-described embodiments are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The flow charts, sequences, elements included in the embodiments, and their arrangements, materials, conditions, shapes, sizes, etc. are not limited to those illustrated, and can be changed as appropriate. In addition, configurations shown in different embodiments can be partially replaced or combined with each other. [Explanation of symbols]

[0045] 1... image processing system, 10... data generation device, 11... processor, 12... storage device, 13... communication IF, 14... input device, 15... output device, 20... diagnostic device, 100... storage unit, 101... display control unit, 102... input unit, 103... generation unit, 104... image processing unit

Claims

1. A display control unit that displays a pathological image and boundaries of a plurality of partial pathological images generated by dividing the pathological image into a plurality of parts in a superimposed manner on a screen; an input unit that receives an input of a label related to pathology to be assigned to each of the plurality of partial pathology images; a generation unit that generates learning data for training a learning model by associating each of the plurality of partial pathology images with a label related to pathology assigned to each of the plurality of partial pathology images; having the input unit stores in a label database a partial pathology image ID that uniquely identifies each of the plurality of partial pathology images and a label related to pathology assigned to each of the plurality of partial pathology images in association with each other; the generation unit acquires the partial pathology image ID and a label related to pathology from the label database, extracts image data of the partial pathology image corresponding to the partial image ID from the pathology images stored in the pathology image database, and generates learning data by combining the image data of the extracted partial pathology image with the label related to pathology corresponding to the partial pathology image ID. Data generation device.

2. The pathology image includes a plurality of pathology images, the input unit accepts an input of information regarding image types corresponding to the plurality of pathological images; A modification unit that modifies pixel values ​​related to colors of the plurality of pathological images according to a modification method determined according to the image type input to the input unit. The data generating device according to claim 1 .

3. The change method includes a method of changing the RGB values ​​of each pixel in the plurality of pathological images to a standardized value, and a method of randomly changing the RGB values ​​of each pixel in the plurality of pathological images. The data generating device according to claim 2 .

4. the display control unit displays both information indicating a ratio for each pathology-related label, which is a target value of the label related to pathology to be assigned to each of the plurality of partial pathology images, and information indicating an actual value of the ratio for each pathology-related label assigned to the plurality of partial pathology images. The data generating device according to any one of claims 1 to 3.

5. the input unit does not accept input of a label related to pathology for a partial pathology image that satisfies a predetermined condition among the plurality of partial pathology images; The generation unit does not include, in the learning data, a partial pathology image for which an input of a label related to pathology has not been accepted, among the plurality of partial pathology images. The data generating device according to any one of claims 1 to 4.

6. the plurality of partial pathology images are pathology images generated by dividing the pathology image at predetermined intervals in the right-left direction and the up-down direction, using a point that is shifted a predetermined distance from a reference point in the pathology image as a reference; The input unit accepts a designation of the predetermined distance. The data generating device according to any one of claims 1 to 5.

7. A data generation method executed by a data generation device, comprising: a step of superimposing a pathological image and boundaries of a plurality of partial pathological images generated by dividing the pathological image into a plurality of partial images and displaying the pathological image on a screen; receiving an input of a label related to pathology to be assigned to each of the plurality of partial pathology images; generating learning data for training a learning model by associating each of the plurality of partial pathology images with a label related to pathology assigned to each of the plurality of partial pathology images; Including, the step of receiving the input includes storing a partial pathology image ID that uniquely identifies each of the plurality of partial pathology images and a label related to pathology assigned to each of the plurality of partial pathology images in a label database in association with each other; The generating step includes acquiring the partial pathology image ID and a label related to pathology from the label database, extracting image data of the partial pathology image corresponding to the partial image ID from the pathology images stored in the pathology image database, and generating learning data by combining the extracted image data of the partial pathology image and the label related to pathology corresponding to the partial pathology image ID. Data generation method.

8. On the computer, a step of superimposing a pathological image and boundaries of a plurality of partial pathological images generated by dividing the pathological image into a plurality of partial images and displaying the pathological image on a screen; receiving an input of a label related to pathology to be assigned to each of the plurality of partial pathology images; generating learning data for training a learning model by associating each of the plurality of partial pathology images with a label related to pathology assigned to each of the plurality of partial pathology images; Run the command, the step of receiving the input includes storing a partial pathology image ID that uniquely identifies each of the plurality of partial pathology images and a label related to pathology assigned to each of the plurality of partial pathology images in a label database in association with each other; The generating step includes acquiring the partial pathology image ID and a label related to pathology from the label database, extracting image data of the partial pathology image corresponding to the partial image ID from the pathology images stored in the pathology image database, and generating learning data by combining the extracted image data of the partial pathology image and the label related to pathology corresponding to the partial pathology image ID. program.

Images