Image generation device, learning device, image generation method, and program

Abstract

In order to solve the problem of minimizing the effect of features of processing in teacher data on the accuracy of machine learning, this image generation device (1) comprises a setting unit (11) for randomly setting a processing parameter, a processing unit (12) for processing a first image and / or a second image according to the processing parameter, and a generation unit (13) for pasting the first image to the second image to generate a third image serving as teacher data for machine learning.

Description

Technical Field

[0001] The present invention relates to a technique for generating an image to be used as teacher data.

Background Art

[0002] Techniques for generating an image to be used as teacher data are known. Such techniques are used, for example, for the purpose of increasing the amount of teacher data and improving the accuracy of machine learning. For example, Non-Patent Document 1 describes a technique for generating an image to be used as teacher data by adding a margin to a lesion region in a medical image as a foreground and pasting it onto another medical image as a background. In this technique, the pixel value of the margin in the foreground is smoothly changed and then pasted onto the medical image as the background.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Non-Patent Document 1 adds a margin along the shape of the lesion area. Therefore, when performing machine learning using the medical image generated by this technique as teacher data, there is a possibility that the features of the margin along the shape of the lesion area and the discontinuity at the boundary of the background are learned. For example, an image recognition model learned using the medical image generated by this technique as teacher data may recognize, as a lesion area, an area similar to the shape of the lesion area and having discontinuity at the boundary of the area, even when the area does not contain a lesion. Thus, there has been a problem that the features of the processing remaining in the teacher data affect the accuracy of machine learning in the teacher data generated by the technique described in Non-Patent Document 1.

[0005] One aspect of the present invention has been made in view of the above problems, and an example of the object is to provide a technique for suppressing the influence of the features of the processing remaining in the teacher data on the accuracy of machine learning.

Means for Solving the Problem

[0006] An image generation apparatus according to one aspect of the present invention includes a setting means for randomly setting processing parameters, a processing means for processing one or both of a first image and a second image according to the processing parameters, and a generation means for generating a third image serving as teacher data for machine learning by pasting the first image onto the second image.

[0007] An image generation method according to one aspect of the present invention includes one or more processors randomly setting processing parameters, processing one or both of a first image and a second image according to the processing parameters, and generating a third image serving as teacher data for machine learning by pasting the first image onto the second image.

[0008] A program according to one aspect of the present invention causes one or more processors to function as setting means for randomly setting processing parameters, processing means for processing one or both of a first image and a second image according to the processing parameters, and generation means for generating a third image serving as teacher data for machine learning by pasting the first image onto the second image.

Advantages of the Invention

[0009] According to one aspect of the present invention, it is possible to suppress the influence of the characteristics of the processing in the teacher data on the accuracy of machine learning.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0011] 〔Exemplary Embodiment 1〕 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0012] <Configuration of Image Generation Device 1> The configuration of the image generation device 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the image generation device 1. As shown in FIG. 1, the image generation device 1 includes a setting unit 11, a processing unit 12, and a generation unit 13. The setting unit 11 is an example of a configuration that realizes the setting means described in the claims. The processing unit 12 is an example of a configuration that realizes the processing means described in the claims. The generation unit 13 is an example of a configuration that realizes the generation means described in the claims.

[0013] The setting unit 11 randomly sets processing parameters. The processing unit 12 processes one or both of the first image and the second image according to the processing parameters. The generation unit 13 generates a third image that serves as teacher data for machine learning by pasting the first image onto the second image.

[0014] <Example of Realization by Program> When the image generation device 1 is realized by a computer, the following program according to this exemplary embodiment is stored in the memory of the computer. The program causes the computer to function as a setting unit 11 that randomly sets processing parameters, a processing unit 12 that processes one or both of the first image and the second image according to the processing parameters, and a generation unit 13 that generates a third image that serves as teacher data for machine learning by pasting the first image onto the second image.

[0015] <Flow of Image Generation Method S1> The image generation device 1 configured as described above executes the image generation method S1 according to this exemplary embodiment. The flow of the image generation method S1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of the image generation method S1. As shown in FIG. 2, the image generation method S1 includes a setting step S11, a processing step S12, and a generation step S13.

[0016] In the setting step S11, the setting unit 11 randomly sets the processing parameters. In the processing step S12, the processing unit 12 processes one or both of the first image and the second image according to the processing parameters. In the generation step S13, the generation unit 13 generates a third image serving as teacher data for machine learning by pasting the first image onto the second image.

[0017] <Effects of the present exemplary embodiment> As described above, in the image generation apparatus 1 and the image generation method S1 according to the present exemplary embodiment, the processing parameters are randomly set, and one or both of the first image and the second image are processed according to the set processing parameters, and the first image is pasted onto the second image to generate a third image serving as teacher data for machine learning. Therefore, according to the present exemplary embodiment, an effect of suppressing the influence of the characteristics of the processing in the teacher data on the accuracy of machine learning can be obtained.

[0018] 〔Exemplary Embodiment 2〕 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. The information processing system 100 according to the present exemplary embodiment is a system that performs machine learning of an image recognition model ML for medical images. When performing machine learning of the image recognition model ML, a medical image including a region indicating a lesion (hereinafter also referred to as a lesion region) is required as teacher data. However, it may be difficult to collect a large number of medical images including the lesion region. For example, in the case of a rare lesion, it is difficult to sufficiently collect such medical images. Therefore, the information processing system 100 generates an extended medical image obtained by extending a medical image including the lesion region, and performs machine learning using the original medical image and the generated extended medical image as teacher data. Hereinafter, in the description of the present exemplary embodiment, components having the same functions as those of the components described in the first exemplary embodiment are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0019] In the present exemplary embodiment, medical images with labels associated with each pixel are used as teacher data. The label takes a value of 1 indicating a lesion, 0 indicating no lesion, or a value between 0 and 1 indicating the degree of a lesion. The image recognition model ML trained using the teacher data is a model that performs a segmentation task of recognizing a lesion area in pixel units.

[0020] <Configuration of Information Processing System 100> The configuration of the information processing system 100 according to the present exemplary embodiment will be described with reference to FIG. 3. FIG. 3 is a block diagram for explaining the configuration of the information processing system 100. As shown in FIG. 3, the information processing system 100 includes an image generation device 1A, a learning device 2, an image storage device 3, and an extended image storage device 4. The image generation device 1A is connected to the image storage device 3 and the extended image storage device 4 so as to be able to read and write information. The learning device 2 is connected to the image storage device 3 and the extended image storage device 4 so as to be able to read information. Note that these devices may be communicably connected via a network.

[0021] The image storage device 3 stores a plurality of medical images IMG1, IMG2, IMG3,... of a predetermined part of the human body. When there is no need to particularly distinguish each of the medical images IMG1, IMG2, IMG3,..., they are simply described as medical images. Examples of the type of medical image include, but are not limited to, endoscopic images, ultrasonic images, CT (Computed Tomography) images, MRI (Magnetic resonance imaging) images, etc. Examples of a predetermined part of the human body include, but are not limited to, the intestine, stomach, esophagus, lungs, head, etc. The plurality of medical images are all medical images of the same type (e.g., endoscopic images) taken of the same predetermined part (e.g., the intestine).

[0022] The plurality of medical images includes a medical image including a lesion region and a medical image not including a lesion region. Also, a label is associated with each medical image. The label is associated with each pixel of the medical image. For example, a label of "1" indicating a lesion is associated with each pixel included in the lesion region, and a label of "0" indicating no lesion is associated with each pixel included in other regions. Each medical image including a lesion region can be an example of the first image described in the claims. Also, each medical image, regardless of whether it includes a lesion region or not, can be an example of the second image. Hereinafter, the first image is described as a foreground image, and the second image is described as a background image.

[0023] The extended image storage device 4 stores the extended medical images Aug-IMG1, Aug-IMG2, Aug-IMG3,... generated by the image generation device 1A. When there is no need to particularly distinguish and explain each of the extended medical images Aug-IMG1, Aug-IMG2, Aug-IMG3,..., each is simply described as an extended medical image. The extended medical image is an example of the third image described in the claims.

[0024] The plurality of extended medical images includes a lesion region. Also, a label determined by the image generation device 1A is associated with each extended medical image. The label is determined and associated with each pixel of the extended medical image.

[0025] The image generation device 1A includes a control unit 110 and a storage unit 120. The control unit 110 comprehensively controls each part of the image generation device 1A. The storage unit 120 stores a program for the control unit 110 to function and various data used by the control unit 110. For example, the storage unit 120 stores constraint information. The constraint information is information indicating constraints regarding the position to be pasted according to a predetermined part included as a subject in the medical image.

[0026] The control unit 110 includes a setting unit 11A, a processing unit 12A, a generation unit 13A, a determination unit 14A, and a pre-processing unit 15A. The determination unit 14A is an example of a configuration that realizes the determination means described in the claims.

[0027] In addition to being configured in the same manner as the setting unit 11, the setting unit 11A is configured as follows. That is, the setting unit 11A randomly sets processing parameters including at least any one of (1) parameters specifying the shape of one or both of the first region and the second region, (2) parameters defining the size of one or both of the first region and the second region, and (3) parameters defining image processing. Examples of image processing include smoothing or transparency. Examples of parameters defining image processing include parameters defining the type of algorithm for performing image processing (e.g., the type of smoothing filter, etc.). Also, other parameters defining image processing include parameters defining the mode of image processing (e.g., the mode of alpha blending). Further, still other parameters defining image processing include parameters defining the intensity of image processing (e.g., the kernel size of the smoothing filter, the transparency of alpha blending). However, the parameters defining image processing are not limited to the examples described above.

[0028] The first region is a region including the boundary of the lesion region associated with the label indicating the lesion in the foreground image. The second region is a region including the boundary of the region where the lesion region overlaps when the foreground image is pasted in the background image. Note that the label indicating the lesion is an example of the "predetermined label" described in the claims. Also, the lesion region is an example of the "predetermined region associated with the predetermined label".

[0029] In addition to being configured in the same manner as the processing unit 12, the processing unit 12A is configured as follows. That is, the processing unit 12A processes one or both of the foreground image and the background image by performing image processing on one or both of the first region and the second region. As the image processing to be performed on one or both of the first region and the second region, the processing unit 12 may perform one of the above-described smoothing and transparency, or both. In this embodiment, it will be described as performing both smoothing and transparency.

[0030] In addition to being configured in the same manner as the generation unit 13, the generation unit 13A is configured as follows. That is, the generation unit 13A determines a label to be associated with the generated extended medical image by referring to the label associated with the foreground image. Further, for example, the generation unit 13A determines a label to be associated with the generated extended medical image by referring to the label associated with the background image and the processing parameters in addition to the label associated with the foreground image.

[0031] The determination unit 14A determines the position to paste the foreground image in the background image by referring to the background image. In addition to referring to the background image, the determination unit 14A further refers to constraint information indicating constraints on the position to be pasted according to a predetermined part, and determines the position to paste the foreground image in the background image. The position to be pasted is hereinafter also referred to as the "paste position".

[0032] The pre-processing unit 15A performs image processing on one or both of the foreground image and the background image. The image processing performed by the pre-processing unit 15A is different from the image processing performed by the processing unit 12A and is executed before the processing unit 12A performs image processing. The image processing performed by the pre-processing unit 15A is also referred to as pre-processing.

[0033] The learning device 2 includes a control unit 210 and a storage unit 220. The control unit 210 controls each part of the learning device 2 in an overall manner. The storage unit 220 stores a program for the control unit 210 to function and various data used by the control unit 210. The control unit 210 includes a learning unit 21. The learning unit 21 learns the image recognition model ML using the teacher data generated using the image generation device 1A. The storage unit 220 stores the image recognition model ML.

[0034] <Flow of the image generation method S1A> The flow of the image generation method S1 executed by the image generation device 1A configured as described above A will be described with reference to FIG. 4. FIG. 4 shows the image generation method S1 AIt is a flowchart for explaining the

[0035] In step S101, the control unit 110 selects and reads a foreground image from the medical images stored in the image storage device 3. Specifically, the control unit 110 selects, as the foreground image, a medical image including a lesion region (in other words, a medical image including pixels associated with a label indicating a lesion).

[0036] In step S102, the preprocessing unit 15A performs preprocessing on the foreground image. Examples of the preprocessing include, but are not limited to, geometric image processing, optical image processing, processing for adding noise, etc. As the preprocessing, various known techniques used for generating other teacher data from the image serving as the teacher data can be adopted. Note that, for example, the preprocessing unit 15A changes the label as necessary in accordance with the preprocessing. Specifically, for example, when the preprocessing is geometric image processing, the preprocessing unit 15A associates the label associated with the pixel before deformation with the pixel after deformation corresponding to the pixel before deformation in accordance with the geometric deformation of the medical image. Such preprocessing is performed based on predetermined parameters.

[0037] In step S103, the control unit 110 selects and reads a background image to be pasted from the medical images stored in the image storage device 3. The background image may be any image different from the foreground image, and may be an image including a lesion region or an image not including a lesion region.

[0038] In step S104, the preprocessing unit 15A performs preprocessing on the background image. Since the details of the preprocessing are the same as those in step S102, a detailed description will not be repeated. Note that the processes in steps S101 to S102 and the processes in steps S103 to S104 may be executed in parallel or in reverse order.

[0039] In step S105, the determination unit 14A determines the paste position of the foreground image in the background image by referring to the background image and the constraint information corresponding to the predetermined part. Further, the determination unit 14A may determine the paste position by referring to the lesion region in the foreground image in addition to the background image and the constraint information. Note that, as described above, the constraint information is information indicating constraints regarding the paste position corresponding to a predetermined part included as a subject in the medical image. The constraint information may be stored in the storage unit 120 or may be stored in an external device.

[0040] (Specific examples of constraint information) Specific examples of the constraint information will be described. Here, it is assumed that an endoscopic image of the intestine is stored in the image storage device 3 as a medical image. That is, the predetermined part is the intestine. In this case, Constraint the information is information indicating constraints regarding the paste position corresponding to the endoscopic image of the intestine. For example, an endoscopic image of the intestine may include a region showing the intestinal wall and a region showing the intestinal tract. Since the region showing the intestinal tract is often photographed as a region with a greater depth than the region showing the intestinal wall, the brightness is often low, and it is less likely that a lesion region is included in such a low-brightness region. Therefore, such a region showing the intestinal tract is not appropriate as a region for pasting the lesion region. Further, the endoscopic image is a rectangular image including a circular region photographed by the endoscope. Therefore, the endoscopic image may include a region indicating outside the photographing range at at least one of the four corners. Such a region indicating outside the photographing range is also not appropriate as a region for pasting the lesion region. Thus, the constraint information includes information indicating a region where it is inappropriate to paste the lesion region (hereinafter, also referred to as a non-pasteable region). The information indicating the non-pasteable region may be a mask image. For example, the mask image may be an image obtained by masking the regions at the four corners of the image indicating outside the photographing range of the endoscope and the region near the center of the image that is likely to show the intestinal tract.

[0041] In addition, there may be a case where an endoscopic image already including a lesion area is used as a background image. In this case, the lesion area is not suitable as an area for pasting other lesion areas. In this case, the information indicating the non-pasteable area may be an image recognition model that recognizes the non-pasteable area. As such an image recognition model, for example, a model that has been separately learned by machine learning can be used.

[0042] The determination unit 14 specifies, for example, a non-pasteable area in the background image with reference to the background image and the constraint information. Further, the determination unit 14 determines the paste position of the foreground image in the background image so that the lesion area in the foreground image is pasted in an area other than the non-pasteable area in the background image. At this time, the determination unit 14 also refers to the shape and size of the lesion area in the foreground image. Note that the determination unit 14 randomly determines the paste position of the lesion area except in the non-pasteable area.

[0043] In step S106, the setting unit 11A determines (1) a parameter that designates the shape of one or both of the first area in the foreground image and the second area in the background image, (2) a parameter that defines the size of one or both of the first area and the second area, and (3) a parameter that defines image processing.

[0044] (Example of setting parameters for defining shape and size) Here, an example of the first region and the second region will be described with reference to FIG. 5. FIG. 5 is a schematic diagram showing an example of the first region, the second region, and an extended medical image. In the example of FIG. 5, the image IMG1 is a medical image selected as the foreground image. In the image IMG1, the region surrounded by the boundary R1a indicates a lesion region. That is, a label indicating a lesion is associated with each pixel inside the boundary R1a. The first region is set as a region including the boundary R1a of the lesion region in the image IMG1. In FIG. 5, the first region is an annular region having the boundary R1a as the inner boundary and the boundary R1b (or R1c, R1d) as the outer boundary. Such an annular first region is also referred to as a margin.

[0045] Also, in FIG. 5, the image IMG2 is a medical image selected as the background image. The second region is set as a region including the boundary R2a. The boundary R2a is the boundary of the region where the lesion region overlaps when the image IMG1 is pasted at the paste position in the image IMG2. In FIG. 5, the second region is an annular region having the boundary R2a as the inner boundary and the boundary R2b (or R2c, R2d) as the outer boundary. Such an annular second region is also referred to as a margin.

[0046] Also, in FIG. 5, an example is shown in which the second region is set as the same region as the region where the first region overlaps. In other words, the boundaries R1b, R1c, R1d in the image IMG1 and the boundaries R2b, R2c, R2d in the image IMG2 overlap when the image IMG1 is pasted at the paste position on the image IMG2. Hereinafter, as shown in FIG. 5, an example where "the second region is the same as the region where the first region overlaps" will be described, and both the first region and the second region will be described as "margin".

[0047] For example, the setting unit 11A may randomly determine any one of a plurality of predetermined shapes as the shape of the margin. Examples of such predetermined shapes include a rectangle such as the boundary R1b, an ellipse such as the boundary R1c, an arbitrary shape such as the boundary R1d, and the like. The arbitrary shape may be, for example, a shape indicated by a boundary R1d connecting points that are separated outward by a randomly set distance from each point on the boundary R1a of the lesion region. In this case, the distance from a certain point on the boundary R1a to the boundary R1d and the distance from another point on the boundary R1a to the boundary R1d may be different. Therefore, the boundary R1d set in this way can be an arbitrary shape surrounded by a curved line with irregularities as shown in FIG. 5. In addition, examples of the predetermined shape types include various polygons, shapes similar to the lesion region, and the like, but are not limited thereto.

[0048] Further, for example, the setting unit 11A may randomly determine any value within a predetermined numerical range as the size of the margin. The parameter to be determined as the size and its numerical range may be predetermined according to the shape of the margin. For example, the parameter to be determined as the size is the length and width if it is a rectangle, the major axis and minor axis if it is an ellipse, the distance from each point on the boundary R1a of the lesion region if it is the above-mentioned arbitrary shape, and the like.

[0049] Further, for example, the setting unit 11A may randomly determine the position of the margin as yet another processing parameter. Since the position of the margin is randomly set, the lesion region does not necessarily have to be included substantially at the center. For example, when the shape of the margin is a rectangle with the positive x-axis direction from left to right and the positive y-axis direction from bottom to top in FIG. 5, this will be described. In this case, the distance d1 from the point on the boundary R1a of the lesion region with the minimum x coordinate to the point on the outer boundary R1b of the margin with the minimum x coordinate and the distance d2 from the point on the boundary R1a with the maximum x coordinate to the point on the boundary R1b with the maximum x coordinate are randomly determined. These do not necessarily have to be equal. The same applies to the y-axis direction. The same also applies to margins of other shapes.

[0050] (Examples of setting parameters defining image processing) Also, specific examples of parameters defining image processing will be described. Here, as described above, the image processing is smoothing or transparency. For example, the setting unit 11A randomly sets, as a processing parameter, the type of smoothing filter for performing smoothing. Specifically, the setting unit 11A may be set to randomly select any one of a plurality of types of smoothing filters (for example, Gaussian filter, moving average filter, median filter, etc.) defined in advance. Also, for example, the setting unit 11A randomly sets a parameter defining the intensity of the set smoothing filter. Specifically, the setting unit 11A may be set by randomly selecting a numerical value within a numerically defined range as the kernel size used in the smoothing filter.

[0051] Also, for example, the setting unit 11A randomly sets the mode of alpha blending for performing transparency. Specifically, the setting unit 11A may be set by randomly selecting any one of a plurality of modes (for example, multiplication, addition, etc.) defined in advance. Also, for example, the setting unit 11A randomly sets a parameter defining the intensity of the set mode. Specifically, the setting unit 11A sets the transparency used in the mode of alpha blending for each pixel of the margin in the foreground image and each pixel of the margin in the background image.

[0052] A specific example of the transparency set for each pixel will be described with reference to FIG. 6. FIG. 6 is a schematic diagram showing an example of the transparency setting. In FIG. 6, the horizontal axis represents the x-axis in the foreground image and the background image, and the vertical axis represents the transparency. The setting unit 11A sets the minimum value (i.e., opaque) as the transparency for each pixel in the lesion area (inside the boundary R1a) in the foreground image. Also, the setting unit 11A sets the maximum value (i.e., transparent) as the transparency for each pixel in the area outside the margin (the first area shown in FIG. 6) in the foreground image (outside the boundary R1b). Further, the setting unit 11A sets the transparency for each pixel in the margin in the foreground image such that it is closer to the minimum value the closer it is to the inner boundary R1a and closer to the maximum value the closer it is to the outer boundary R1b, that is, the transparency is set to change from opaque to transparent from the inside to the outside.

[0053] Also, the setting unit 11A sets the maximum value (i.e., transparent) as the transparency for each pixel in the area where the lesion overlaps (inside the boundary R2a) in the background image. Also, the setting unit 11A sets the minimum value (i.e., opaque) as the transparency for each pixel in the area outside the margin (the second area shown in FIG. 6) in the background image (outside the boundary R2b). Further, the setting unit 11A sets the transparency for each pixel in the margin in the background image such that it is closer to the maximum value the closer it is to the inner boundary R2a and closer to the minimum value the closer it is to the outer boundary R2b, that is, the transparency is set to change from transparent to opaque from the inside to the outside.

[0054] In step S107, the processing unit 12A performs image processing on the foreground image using the processing parameters randomly set in step S106. For example, the processing unit 12A performs smoothing on the margin in the foreground image using the set type of smoothing filter and kernel size. Also, the processing unit 12A calculates pixel values for alpha blending with the background image based on the transparency set for the foreground image.

[0055] In step S108, the processing unit 12A performs image processing on the background image using the processing parameters randomly set in step S106. Since specific examples of the image processing to be performed are the same as those in step S107, detailed descriptions will not be repeated. Note that the processes in steps S107 to S108 may be executed in parallel or in reverse order.

[0056] In step S109, the processing unit 12A performs image processing on the foreground image and the background image to match their respective image qualities. Examples of the image processing for matching the image quality include, but are not limited to, color correction processing for matching the color tone, blur processing for matching the blur degree, and the like. For example, the processing unit 12A refers to the color tone of the foreground image and the color tone of the background image, and performs color correction processing for matching them on the foreground image and the background image. Also, for example, the processing unit 12A refers to the blur degree of the foreground image and the blur degree of the background image, and performs blur processing for matching them on the foreground image and the background image.

[0057] In step S110, the generation unit 13A generates an extended medical image by executing a process of pasting the processed foreground image onto the processed background image. The pasting process may be, for example, a process of adding each pixel value of the processed foreground image and each pixel value of the processed background image. The extended medical image will be described with reference to FIG. 5. For example, the extended medical image Aug-IMG1 is generated by pasting the image IMG1 after image processing using the margin defined by the boundary R1b onto the image IMG2 after image processing using the margin defined by the boundary R2b. Also, for example, the extended medical image Aug-IMG2 is generated by pasting the image IMG1 after image processing using the margin defined by the boundary R1c onto the image IMG2 after image processing using the margin defined by the boundary R2c. For example, the extended medical image Aug-IMG3 is generated by pasting the image IMG1 after image processing using the margin defined by the boundary R1d onto the image IMG2 after image processing using the margin defined by the boundary R2d. As shown in this example, different extended medical images Aug-IMG1 to Aug-IMG3 are generated from the same images IMG1 and IMG2 according to randomly set processing parameters.

[0058] In step S111, the generation unit 13A determines the label associated with each pixel of the extended medical image by referring to the label associated with the foreground image, the label associated with the background image, and the processing parameters.

[0059] A specific example of the process for determining labels will be described. Hereinafter, the area on the extended medical image where the lesion area in the foreground image is pasted will be referred to as the lesion area in the extended medical image. Also, the area on the extended medical image where the margin in the foreground image is pasted will be referred to as the margin in the extended medical image. Further, the area on the extended medical image corresponding to the outside of the margin in the background image will be referred to as the outside of the margin in the extended medical image. As an example, the generation unit 13A determines, as the label to be associated with each pixel in the lesion area of the extended medical image, the label associated with each pixel in the lesion area of the foreground image. Also, the generation unit 13A determines, as the label to be associated with each pixel outside the margin in the background image, the label associated with each pixel outside the margin on the extended medical image. Also, the generation unit 13A determines, as the label to be associated with each pixel in the margin of the extended medical image, a label corresponding to the transparency set for the margin in the foreground image. Specifically, for each pixel, the generation unit 13A calculates a value between 0 and 1 as the label such that the smaller the transparency (the closer to opaque), the closer to "1" indicating a lesion, and the larger the transparency (the closer to transparent), the closer to "0" indicating no lesion.

[0060] As a result, a label indicating a lesion is associated with each pixel in the lesion area of the extended medical image. Also, a label approaching from 1 to 0 from the inside to the outside is associated with each pixel in the margin of the extended medical image. Also, the label associated with the original background image is associated with each pixel outside the margin in the extended medical image.

[0061] Also, the generation unit 13A stores the extended medical image with the labels associated therewith in the extended image storage device 4. The extended medical image includes the lesion area and serves as positive teacher data for training the image recognition model ML.

[0062] The image generation device 1A repeats the processes of steps S101 to S111 until a predetermined end condition is satisfied. As a result, a plurality of extended medical images are generated and stored in the extended image storage device 4. The predetermined end condition may be, but is not limited to, that the number of generated extended medical images exceeds a threshold value, the processing time exceeds a threshold value, and the like.

[0063] The learning unit 21 in the learning device 2 learns the image recognition model ML using the medical images stored in the image storage device 3 and the extended medical images stored in the extended image storage device 4.

[0064] <Effects of this exemplary embodiment> In this exemplary embodiment, in addition to the same configuration as in the first exemplary embodiment, a configuration is adopted in which the label associated with the extended medical image is determined by referring to the label associated with the foreground image. As a result, when the foreground image includes a lesion area, the generated extended medical image can be used as positive teacher data for the lesion area.

[0065] Further, in this exemplary embodiment, image processing is performed on one or both of the margin including the boundary of the lesion area in the foreground image and the margin including the boundary of the area where the lesion area overlaps when the foreground image is pasted in the background image, thereby processing one or both of the foreground image and the background image. As a result, it is possible to set a margin, which is an area for reducing the influence of the processing remaining in the extended medical image.

[0066] In addition, in the present exemplary embodiment, in addition to the label associated with the foreground image, the label associated with the background image and the randomly set processing parameters are further referred to, and a configuration is adopted in which the label associated with the enhanced medical image is determined. As a result, for example, in the margin area of the enhanced medical image, labels that change from the label indicating a lesion to the label indicating no lesion can be associated from the inside to the outside. As a result, it is possible to reduce the learning of the margin features as the lesion features.

[0067] In addition, in the present exemplary embodiment, the processing parameters include at least any one of (1) a parameter specifying the shape of the margin, (2) a parameter defining the size of one or both sides of the margin, and (3) a parameter defining image processing. As a result, when generating a plurality of teacher data, the margins in the plurality of teacher data are distributed uniformly in terms of their shape and size without any tendency. Also, the traces of the image processing applied to the margins in the plurality of teacher data are uniformly distributed in terms of their features without any tendency. As a result, it is possible to reduce the learning of the margin features as the lesion features.

[0068] In addition, in the present exemplary embodiment, a configuration is adopted in which the image processing applied to the margin is smoothing or transparency. As a result, the margin can be smoothed and made transparent, and it is possible to reduce the learning of the margin features as the lesion features.

[0069] In addition, in the present exemplary embodiment, a configuration is adopted in which the position where the foreground image is pasted on the background image is determined by referring to the background image. As a result, the foreground image can be pasted at an appropriate position according to the background image.

[0070] Also, in the present exemplary embodiment, the foreground image and the background image are medical images obtained by photographing a predetermined part of the human body, and the position where the foreground image is pasted on the background image is determined by further referring to constraint information indicating constraints regarding the pasting position according to the predetermined part. Thereby, the foreground image, which is a medical image, can be pasted at an appropriate position of the background image, which is also a medical image.

[0071] [Modification Example 1] In the above-described exemplary embodiment 2, in addition to generating teacher data with a lesion area pasted thereon, the image generation apparatus 1A can be modified to generate teacher data with a non-lesion dummy area pasted thereon. In this modification example, the margin in the foreground image is an area including the boundary of the dummy area to which a label indicating no lesion is associated. Note that, as the margin in the background image, the same area as the area where the margin in the foreground image overlaps is adopted, as in exemplary embodiment 2. In this modification example, the following steps in the image generation method S1A are modified.

[0072] In step S101, the control unit 110 selects any one of the medical images stored in the image storage device 3 as the foreground image. The foreground image may or may not include a lesion area.

[0073] In step S105, the setting unit 11A operates substantially the same as step S105 described above, but differs in the following point. The setting unit 11A randomly determines a dummy area to which a label indicating that there is no lesion in the foreground image is associated. The dummy area only needs to be associated with a label indicating that there is no lesion, and the position, shape, and size may be randomly determined.

[0074] In step S111, the generation unit 13A operates in substantially the same manner as step S111 described above, but differs in the following points. The generation unit 13A determines "0", which indicates that there is no lesion, as the label associated with each pixel in the dummy region and the margin in the extended medical image. Further, the generation unit 13A determines the label associated with each pixel outside the margin in the background image as the label associated with each pixel outside the margin in the extended medical image.

[0075] Except for the above points, the image generation method S1A according to this modification example is similarly described by replacing "lesion area" with "dummy area" in the description of the image generation method S1A according to the above-described exemplary embodiment 2. Thereby, in this modification example, in step S111, an extended medical image that does not include a lesion area is generated as negative teacher data.

[0076] The extended medical image generated in this modification example will be described with reference to FIG. 7. FIG. 7 is a schematic diagram showing an example of an extended medical image to which a dummy area is attached. In the example of FIG. 7, an image IMG3 that does not include a lesion area is selected as the foreground image. Further, in the image IMG3, a dummy area surrounded by the boundary R1e is set. Further, in the image IMG3, a margin that includes the boundary R1e as the inner boundary and the boundary R1f as the outer boundary is set. Further, an image IMG4 that does not include a lesion area is selected as the background image. Further, in the image IMG4, a margin that includes the boundary R2e as the inner boundary and the boundary R2f as the outer boundary is set. The boundary R1e and the boundary R2e overlap when the foreground image is pasted at the paste position on the background image. Also, the boundary R1f and the boundary R2f overlap when the foreground image is pasted at the paste position on the background image. After image processing is performed on the image IMG3 according to the processing parameters and image processing is performed on the image IMG4 according to the processing parameters, the image IMG3 is pasted onto the image IMG4, thereby generating an extended medical image Aug-IMG4.

[0077] Each pixel of the augmented medical image Aug-IMG4 is associated with a label 0 indicating that it is not a lesion. Such an augmented medical image Aug-IMG4 may include discontinuity features at the margin, but can be used as negative ground truth data indicating that it is not a lesion. By using this modification in addition to Exemplary Embodiment 2, the augmented image storage device 4 stores negative ground truth data in addition to positive ground truth data.

[0078] In addition to being configured in the same manner as in Exemplary Embodiment 2, the learning unit 21 in the learning device 2 uses an augmented medical image with a dummy region pasted thereon to train the image recognition model ML. Both the positive ground truth data and the negative ground truth data may include discontinuity features near the boundary of the lesion region (or dummy region). Therefore, it is possible to reduce the possibility that the discontinuity feature is learned as a feature of the lesion, and improve the accuracy of the image recognition model ML.

[0079] According to this modification, one or both of the margins including the boundary of the dummy region where no label indicating a lesion is associated in the foreground image and the margin including the boundary of the region where the dummy region overlaps when the foreground image is pasted in the background image are subjected to image processing, thereby processing one or both of the foreground image and the background image. As a result, an augmented medical image that includes a margin with discontinuity and does not include a lesion region inside the margin can be generated as negative ground truth data. As a result, it is possible to reduce the possibility that the discontinuity feature is learned as a feature indicating a lesion.

[0080] [Modification 2] In the above-described Exemplary Embodiment 2, the processing unit 12A may perform image processing on the region inside the margin. The region inside the margin is the lesion region in the foreground image and the region where the lesion region overlaps in the background image. In this modification, the following steps in the image generation method S1A are modified.

[0081] In step S105, the setting unit 11A operates in substantially the same manner as step S105 described above, but there are the following differences. In the foreground image, the setting unit 11A randomly sets, in addition to the processing parameters for the margin, the processing parameters for the lesion area. Also, in the background image, in addition to the processing parameters for the margin, , disease the setting unit 11A randomly sets the processing parameters for the area where the lesion areas overlap.

[0082] A specific example of the transparency set for each pixel in the lesion area and the area where the lesion areas overlap in this modified example will be described with reference to FIG. 8. FIG. 8 is a schematic diagram showing an example of transparency setting. In FIG. 8, the horizontal axis represents the x-axis in the foreground image and the background image, and the vertical axis represents the transparency. In the foreground image, the setting unit 11A sets a semi-transparent transparency (e.g., 50%) for each pixel in the lesion area (inside the boundary R1a). Also, in the foreground image, for each pixel in the area outside the margin (outside the boundary R1b), the maximum value (i.e., transparent) is set as the transparency. Further, in the foreground image, for each pixel in the margin, the transparency is set such that it is closer to 50% the closer it is to the inner boundary R1a and closer to the maximum value the closer it is to the outer boundary R1b, that is, the transparency changes from semi-transparent to transparent from the inside to the outside.

[0083] Also, in the background image, the setting unit 11A sets a semi-transparent transparency (e.g., 50%) for each pixel in the area where the lesion areas overlap (inside the boundary R2a). Also, in the background image, for each pixel in the area outside the margin (outside the boundary R2b), the minimum value (i.e., opaque) is set as the transparency. Further, in the background image, for each pixel in the margin, the transparency is set such that it is closer to 50% the closer it is to the inner boundary R2a and closer to the minimum value the closer it is to the outer boundary R2b, that is, the transparency changes from semi-transparent to opaque from the inside to the outside.

[0084] In step S111, the generation unit 13A operates in substantially the same manner as step S111 described above, but there are the following differences. The generation unit 13A sets, as a label associated with each pixel in the lesion region in the extended medical image, a value obtained by multiplying the label indicating the lesion by the transparency. For example, if the transparency set for each pixel in the lesion region is 50%, the label becomes 0.5. Further, the generation unit 13A determines, as the label associated with each pixel outside the margin in the background image, the label associated with each pixel outside the margin in the extended medical image. Further, the generation unit 13A determines, as the label associated with each pixel in the margin of the extended medical image, a label corresponding to the transparency set for the margin in the foreground image.

[0085] As a result, for example, the label "0.5" is associated with each pixel in the lesion region in the extended medical image. Also, for each pixel in the margin of the extended medical image, a label that approaches 0 from 0.5 from the inside to the outside is associated. Further, for each pixel outside the margin in the extended medical image, the label associated with the original background image is associated.

[0086] According to this modification example, in addition to the margins of the foreground image and the background image, a configuration is adopted in which image processing is performed on the lesion region in the foreground image and the region where the lesion region overlaps in the background image. As a result, in the generated extended medical image, the difference in feature amounts between the lesion region and the region outside thereof can be further reduced, and the discontinuity can be further alleviated.

[0087] Note that this modification example can also be combined with Modification Example 1. That is, the processing unit 12A may perform image processing on the dummy region in the foreground image and the region where the dummy region overlaps in the background image, in addition to the margins of the foreground image and the background image.

[0088] 〔Other Modification Examples〕 Also, in the above-described exemplary embodiment 2, an example of performing smoothing or transparency as image processing was described. However, not limited thereto, the processing unit 12A may perform other image processing on one or both of the foreground image and the background image instead of or in addition to smoothing or transparency.

[0089] Also, in the above-described exemplary embodiment, when performing a plurality of types of image processing on one or both of the foreground image and the background image, a margin may be set according to the type of image processing. For example, the margin for performing transparency and the margin for performing a smoothing filter may differ in at least one of shape, size, and position.

[0090] Also, in the above-described exemplary embodiment 2, the margin in the background image was described as being the same as the area overlapping the margin in the foreground image. However, these do not necessarily have to be the same. For example, the margin in the background image may include the margin in the foreground image and be a wider area than the margin in the foreground image.

[0091] Also, in the above-described exemplary embodiment 2, the margin in the foreground image was described as including the boundary of the lesion area as the inner boundary, that is, being the area outside the lesion area. However, not limited thereto, the margin in the foreground image may include the area inside the lesion area. The same applies to the margin in the background image.

[0092] Also, in the above-described exemplary embodiment 2, an example where the image processing performed on the foreground image and the image processing performed on the background image are the same was described. However, not limited thereto, the image processing performed on the foreground image and the image processing performed on the background image may be at least partially different. For example, smoothing and transparency may be performed on the foreground image, and transparency may be performed on the background image without performing smoothing.

[0093] Further, the above-described exemplary embodiment 2 has described an example in which the types identified by the label are two types, i.e., whether or not it is a lesion. However, the types identified by the label are not limited to this, and may be three or more types (for example, a first type of lesion, a second type of lesion, and no lesion, etc.). In this case, the generation unit 13A may determine the label associated with each pixel of the margin in the extended medical image as follows. For example, assume that the first label is associated with the area inside the margin in the foreground image, and the second label is associated with the area outside the margin in the background image. As an example, instead of associating one label with each pixel of the margin in the extended medical image, the generation unit 13A may associate the weight for the first label and the weight for the second label. Such weights are set according to the transparency set for each pixel of the margin in the foreground image.

[0094] Further, the above-described exemplary embodiment 2 has been described as generating teacher data used for machine learning of an image recognition model that performs a segmentation task. However, the exemplary embodiment 2 is not limited to this, and is also applicable when generating teacher data used for machine learning that performs a detection task of detecting a rectangular area including a lesion, a classification task of recognizing the class of a lesion in image units, etc.

[0095] In the case of the classification task, a label is associated with the medical image in image units. Therefore, the generation unit 13A determines the label associated with the foreground image as the label to be associated with the generated extended medical image. Also, in the case of the detection task, the medical image, which is the teacher data, is associated with the information of the rectangular area including the lesion and the label corresponding to each rectangular area. Therefore, the generation unit 13A determines the label associated with the rectangular area including the lesion area in the extended medical image as the label associated with the rectangular area including the lesion area in the foreground image. Further, for the area outside the margin in the extended medical image, the generation unit 13A associates the rectangular area and its label associated with the area outside the margin in the background image.

[0096] In addition, the above-described exemplary embodiment 2 is also applicable when generating teacher data using images other than medical images. For example, this exemplary embodiment can generate teacher data for recognizing a target area in an image where the difference in feature amounts between the area of the target to be recognized and the area that is not the target is small, in other words, an image in which it is difficult to distinguish between the area of the target to be recognized and the area that is not the target. In such an image, it is particularly important to reduce the learning of the discontinuity feature at the boundary between the target area in the foreground image and the background image as a target feature. As an example of such an image and the target to be recognized, there is an example of recognizing an object (car, person, forest, farmland, road, etc.) in an aerial photograph. The aerial photograph may be taken by a visible light camera, or may be taken by a radar mounted on an artificial satellite or an aircraft, etc. Since an image taken by a radar is generated by converting the radar reflection into a two-dimensional image, it tends to be even more difficult to distinguish an object from its surroundings compared to an image taken by a visible light camera. However, the types of images other than medical images applicable in this exemplary embodiment are not limited to the above-described examples.

[0097] 〔Example of Realization by Software〕 Some or all of the functions of the image generation apparatuses 1, 1A, and the learning apparatus 2 (hereinafter also referred to as apparatuses) may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0098] In the latter case, the above device is realized by, for example, a computer that executes program instructions of software that realizes each function of the device. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 9. Computer C includes at least one processor C1 and at least one memory C2. A program P for operating computer C as each device is recorded in memory C2. In computer C, processor C1 reads and executes program P from memory C2, whereby each function of the above device is realized.

[0099] As processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. As memory C2, for example, a flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0100] Note that computer C may further include a RAM (Random Access Memory) for expanding program P during execution and temporarily storing various data. Also, computer C may further include a communication interface for transmitting and receiving data to and from other devices. Also, computer C may further include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0101] Also, the program P can be recorded on a non-transitory tangible recording medium M readable by the computer C. As such a recording medium M, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit can be used. The computer C can acquire the program P via such a recording medium M. Also, the program P can be transmitted via a transmission medium. As such a transmission medium, for example, a communication network or a broadcast wave can be used. The computer C can also acquire the program P via such a transmission medium.

[0102] [Supplementary Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0103] [Supplementary Note 2] Some or all of the above-described embodiments may also be described as follows. However, the present invention is not limited to the aspects described below.

[0104] (Supplementary Note 1) Setting means for randomly setting processing parameters; Processing means for processing one or both of the first image and the second image according to the processing parameters; Generating means for generating a third image serving as teacher data for machine learning by pasting the first image onto the second image, An image generation device.

[0105] (Supplementary Note 2) The generating means determines a label to be associated with the third image by referring to a label associated with the first image. The image generation device according to Supplementary Note 1.

[0106] (Supplementary Note 3) The generation means determines the label to be associated with the third image by referring to the label associated with the second image and the processing parameters in addition to the label associated with the first image. The image generation device according to Supplementary Note 2.

[0107] (Supplementary Note 4) The processing means processes one or both of the first image and the second image by performing image processing on one or both of a first region including the boundary of a predetermined region associated with a predetermined label in the first image and a second region including the boundary of a region where the predetermined region overlaps when the first image is pasted in the second image. The image generation device according to any one of Supplementary Notes 1 to 3.

[0108] (Supplementary Note 5) The processing means processes one or both of the first image and the second image by performing image processing on one or both of a first region including the boundary of a dummy region not associated with a predetermined label in the first image and a second region including the boundary of a region where the dummy region overlaps when the first image is pasted in the second image. The image generation device according to any one of Supplementary Notes 1 to 3.

[0109] (Supplementary Note 6) The processing parameters include at least any one of (1) a parameter specifying the shape of one or both of the first region and the second region, (2) a parameter defining the size of one or both of the first region and the second region, and (3) a parameter defining the image processing. The image generation device according to Supplementary Note 4 or 5.

[0110] (Supplementary Note 7) The image processing is smoothing or transparency. The image generation device according to any one of Supplementary Notes 4 to 6.

[0111] (Supplementary Note 8) Determination means for determining, by referring to the second image, the position where the first image is to be pasted in the second image, and further comprising The image generation device according to any one of Appendices 1 to 7.

[0112] (Appendix 9) The first image and the second image are medical images obtained by photographing a predetermined part of the human body, The determination means determines the position where the first image is to be pasted in the second image by further referring to constraint information indicating constraints regarding the position to be pasted according to the predetermined part. The image generation device according to Appendix 8.

[0113] (Appendix 10) Learning means for learning an image recognition model using teacher data generated using the image generation device according to any one of Appendices 1 to 9. A learning device including

[0114] (Appendix 11) One or more processors Randomly setting processing parameters; Processing one or both of the first image and the second image according to the processing parameters; Generating a third image serving as teacher data for machine learning by pasting the first image into the second image; An image generation method including

[0115] (Appendix 12) One or more processors Setting means for randomly setting processing parameters; Processing means for processing one or both of the first image and the second image according to the processing parameters; Generating means for generating a third image serving as teacher data for machine learning by pasting the first image into the second image; A program that functions as

[0116] [Supplementary Note 3] Some or all of the above-described embodiments can also be expressed as follows.

[0117] (Supplementary Note 13) An image generation device including at least one processor, the processor performing: a setting process of randomly setting processing parameters; a processing process of processing one or both of a first image and a second image according to the processing parameters; a generation process of generating a third image serving as teacher data for machine learning by pasting the first image onto the second image.

[0118] Note that this image generation device may further include a memory, and the memory may store a program for causing the processor to execute the setting process, the processing process, and the generation process. Further, this program may be recorded on a non-transitory tangible computer-readable recording medium.

[0119] (Supplementary Note 14) A learning device including at least one processor, the processor performing a learning process of training an image recognition model using teacher data generated by using the image generation device described in Supplementary Note 13.

[0120] Note that this learning device may further include a memory, and the memory may store a program for causing the processor to execute the learning process. Further, this program may be recorded on a non-transitory tangible computer-readable recording medium.

Explanation of Reference Numerals

[0121] 100 Information processing system 1, 1A Image generation device 2 Learning device 3 Image storage device​​ 4 Extended Image Memory Device 11, 11A Setting Unit 12, 12A Processing Unit 13, 13A Generation Unit 14, 14A Decision Unit 15A Pre-Processing Unit 21 Learning Unit 110, 210 Control Unit 120, 220 Memory Unit C1 Processor C2 Memory

Claims

1. A setting means for randomly setting processing parameters; a processing means for processing one or both of the first image and the second image in accordance with the processing parameters; and a generating means for generating a third image serving as training data for machine learning by pasting the first image onto the second image. Image generating device.

2. The generating means determines a label to be associated with the third image by referring to a label associated with the first image.

2. The image generating device of claim 1.

3. the generating means determines a label to be associated with the third image by referring to a label associated with the second image and the processing parameters in addition to the label associated with the first image; 3. The image generating device of claim 2.

4. the processing means processes one or both of the first image and the second image by performing image processing on one or both of a first region including a boundary of a predetermined region associated with a predetermined label in the first image and a second region including a boundary of a region in the second image with which the predetermined region overlaps when the first image is pasted; 3. An image generating device according to claim 1 or 2.

5. the processing means processes one or both of the first image and the second image by performing image processing on one or both of a first region including a boundary of a dummy region in the first image that is not associated with a predetermined label, and a second region including a boundary of a region in the second image that will overlap with the dummy region when the first image is pasted; 3. An image generating device according to claim 1 or 2.

6. The processing parameters include at least any of: (1) a parameter that specifies a shape of one or both of the first region and the second region; (2) a parameter that specifies a size of one or both of the first region and the second region; and (3) a parameter that specifies the image processing.

5. The image generating device of claim 4.

7. The image processing is smoothing or transparency.

5. The image generating device of claim 4.

8. and determining means for determining a position in the second image to paste the first image by referring to the second image.

3. An image generating device according to claim 1 or 2.

9. the first image and the second image are medical images of a predetermined part of a human body; the determining means determines a position at which the first image is to be pasted in the second image by further referring to constraint information indicating constraints on the position at which the first image is to be pasted according to the predetermined portion.

9. The image generating device of claim 8.

10. A learning means for learning an image recognition model using training data generated by the image generating device according to claim 1 or 2; A learning device comprising:

11. One or more processors Randomly setting processing parameters; processing one or both of the first image and the second image according to the processing parameters; generating a third image serving as training data for machine learning by pasting the first image onto the second image; 13. An image generating method comprising:

12. One or more processors, A setting means for randomly setting processing parameters; a processing means for processing one or both of the first image and the second image in accordance with the processing parameters; A generating means for generating a third image serving as training data for machine learning by pasting the first image onto the second image; A program that functions as a

Images