Method for generating training data, program and apparatus thereof

Abstract

To provide a learning data generation method capable of reducing working hours of annotation by manual work.SOLUTION: A learning data generation method generates second multiple models to which a function of geometrically changing an object is imparted in a first model for generating a first pseudo image on the basis of a first image including the predetermined object by allowing the changing method to differ, generates the first pseudo image in the first model from the first image, generates second multiple pseudo images in each of the second multiple models from the first pseudo image, obtains the amount of change between the second pseudo image and the first pseudo image for each of the second multiple pseudo images, generates a label of the first pseudo image on the basis of the respective obtained amounts of change, generates learning data by correlating the first pseudo image with the label, and generates a learning data set by repeating each processing from generation of the first pseudo image to generation of the label for multiple times.SELECTED DRAWING: Figure 7

Description

Technical Field

[0004] , , , , , , ,

[0001] The present invention relates to a learning data generation method, a learning data generation program, and a learning data generation device for generating a learning data set used for machine learning a machine learning model for detecting an object.

Background Art

[0002] In recent years, machine learning has been researched and developed and is being applied to various fields. This machine learning requires a relatively large number of learning data for performing the machine learning. In particular, in supervised machine learning, it is necessary to attach teacher data (teacher label, label) indicating whether the learning data is correct or not, that is, the teacher, to the learning data. Therefore, it is necessary to generate teacher data (annotation) for each learning data, which causes a great deal of man-hours. For this reason, reduction of the man-hours is desired, and for example, there is a technique disclosed in Non-Patent Document 1.

[0003] The machine learning method disclosed in Non-Patent Document 1 machine-learns the correspondence between domains in two image data sets with different domains by an adversarial generative network (GAN, Generative Adversarial Network), rather than the correspondence between pixels of paired images. In this machine learning method, since two image data sets are machine-learned with a cyclic structure of conversion and inverse conversion, machine learning can be performed without preparing a large amount of image data sets as learning data sets.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] The machine learning method disclosed in Non-Patent Document 1 does not require a large image dataset as a training dataset, but this does not mean that it is completely unnecessary; a small amount of training dataset is required, and annotation is necessary accordingly.

[0006] This invention was made in view of the above circumstances, and its purpose is to provide a training data generation method, a training data generation program, and a training data generation apparatus that can further reduce the amount of manual annotation required. [Means for solving the problem]

[0007] As a result of various studies, the inventors have found that the above objective can be achieved by the present invention as follows. That is, a method for generating training data according to one aspect of the present invention comprises: a first step of generating multiple second models, each with a different method of change, which are provided with a function to geometrically change the object in a first model that generates a first pseudo-image based on a first image containing a predetermined object; a second step of generating a first pseudo-image from the first image using the first model; a third step of generating multiple second pseudo-images from the first pseudo-image using each of the multiple second models; a fourth step of determining the amount of change between each of the multiple second pseudo-images and the first pseudo-image, and generating a label for the first pseudo-image based on the amount of change determined for each of the multiple second pseudo-images; a fifth step of generating training data by associating the first pseudo-image with the label; and a sixth step of generating multiple training data as a training dataset by repeating the second to fifth steps multiple times.

[0008] This method of generating training data allows for the generation of labels for the first pseudo-images generated by the first model by using a second model generated based on the first model. This eliminates the need for annotation, further reducing the manual effort required for annotation.

[0009] In another embodiment, the learning data generation method described above comprises a 41st step of generating change amount data by determining the change amount between each of the plurality of second pseudo-images and the first pseudo-image; a 42nd step of generating complementary change amount data by filling in the missing values ​​in each of the change amount data obtained in the 41st step for each of the plurality of second pseudo-images; and a 43rd step of generating the labels based on each of the complementary change amount data generated in the 42nd step for each of the change amount data.

[0010] This method of generating training data compensates for missing change data, allowing for more accurate label generation.

[0011] In another embodiment, the above-described method for generating training data further includes a seventh step prior to the first step, in which the first model is STYLEGAN or STYLEGAN2, and the first model is trained using a training dataset for the first model comprising a plurality of first images and class labels associated with each of the plurality of first images.

[0012] This method of generating training data allows the first model to be generated using machine learning.

[0013] In another embodiment, in the above-described methods for generating training data, the way of change includes at least two of the following: a first way of change in which the object is moved in the left-right direction; a second way of change in which the object is moved in the up-down direction; a third way of change in which the object is moved in the diagonal direction; and a fourth way of change in which the object is either enlarged or reduced, and the first step generates at least two second models by performing at least two of the above.

[0014] According to this, a method for generating training data that includes at least two of the first to third ways of change can be provided.

[0015] In another embodiment, in the above-described training data generation method, the change between the first pseudo-image and the second pseudo-image is represented by optical flow.

[0016] According to this, a method for generating training data is provided in which the amount of change between the first pseudo-image and the second pseudo-image is represented by optical flow.

[0017] A learning data generation program according to another aspect of the present invention is a program that causes a computer to execute: a first step of generating multiple second models, each with a different method of change, which are added to a first model that generates a first pseudo-image based on a first image containing a predetermined object, and which has the function of geometrically changing the object; a second step of generating a first pseudo-image from the first image using the first model; a third step of generating multiple second pseudo-images from the first pseudo-image using each of the multiple second models; a fourth step of determining the amount of change between each of the multiple second pseudo-images and the first pseudo-image, and generating a label for the first pseudo-image based on the amount of change determined for each of the multiple second pseudo-images; a fifth step of generating learning data by associating the first pseudo-image with the label; and a sixth step of generating multiple learning data as a learning dataset by repeating the second to fifth steps multiple times.

[0018] Such a training data generation program can generate labels for the first pseudo-images generated by the first model by using a second model generated based on the first model. This eliminates the need for annotation, further reducing the manual effort required for annotation.

[0019] A learning data generation device according to another aspect of the present invention includes: a second model generation unit that generates a plurality of second models, each having a function to geometrically change the object, to a first model that generates a first pseudo-image based on a first image including a predetermined object, with the method of change being different; a first pseudo-image generation unit that performs a first pseudo-image generation process to generate a first pseudo-image from the first image using the first model; a second pseudo-image generation unit that performs a second pseudo-image generation process to generate a plurality of second pseudo-images from the first pseudo-image using each of the plurality of second models; and for each of the plurality of second pseudo-images, the second pseudo-image The system includes: a label generation unit that calculates the amount of change between the first pseudo-image and the first pseudo-image and performs a label generation process that generates a label for the first pseudo-image based on the amount of change calculated for each of the plurality of second pseudo-images; a learning data generation unit that performs a learning data generation process that generates learning data by associating the first pseudo-image with the label; and a learning dataset generation unit that generates a plurality of learning datasets by repeating the first pseudo-image generation process, the second pseudo-image generation process, the label generation process and the learning data generation process multiple times.

[0020] Such a training data generation device can generate labels for the first pseudo-image generated by the first model by using a second model generated based on the first model, thus eliminating the need for annotation and further reducing the amount of manual annotation work required. [Effects of the Invention]

[0021] The learning data generation method, learning data generation program, and learning data generation apparatus according to the present invention can further reduce the amount of manual annotation required.

Brief Description of the Drawings

[0022] [Figure 1] It is a block diagram showing the configuration of the learning data generation device in the embodiment. [Figure 2] As an example, it is a diagram showing the second learning data. [Figure 3] As an example, it is a diagram showing the first pseudo-image. [Figure 4] As an example, it is a diagram showing the optical flow. [Figure 5] As an example, it is a diagram showing the change amount map visualizing the change amount and its complementary change amount map. [Figure 6] As an example, it is a diagram showing the first pseudo-image and its heat map. [Figure 7] It is a flowchart showing the operation of the learning data generation device.

Modes for Carrying Out the Invention

[0023] Hereinafter, one or more embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the disclosed embodiments. In each figure, components with the same reference numerals indicate the same components, and the description thereof will be omitted as appropriate. In this specification, when referring to components generically, reference numerals without suffixes are used, and when referring to individual components, reference numerals with suffixes are used.

[0024] The learning data generation device in this embodiment is a device that generates a learning dataset comprising multiple supervised learning datasets for machine learning a machine learning model that detects a predetermined object. This learning data generation device comprises a second model generation unit, a first pseudo-image generation unit, a second pseudo-image generation unit, a label generation unit, a learning data generation unit, and a learning dataset generation unit. The second model generation unit generates multiple second models, each with a different method of transformation, which are added to the first model that generates a first pseudo-image based on a first image containing a predetermined object, by providing the function of geometrically changing the object. The first pseudo-image generation unit performs a first pseudo-image generation process that generates a first pseudo-image from the first image using the first model. The second pseudo-image generation unit performs a second pseudo-image generation process that generates multiple second pseudo-images from the first pseudo-image using each of the multiple second models. The label generation unit calculates the change between each of the plurality of second pseudo-images and the first pseudo-image, and performs a label generation process to generate a label for the first pseudo-image based on the change calculated for each of the plurality of second pseudo-images. The training data generation unit performs a training data generation process to generate training data by associating the first pseudo-image with the label. The training dataset generation unit generates multiple training data as a training dataset by repeating the first pseudo-image generation process, the second pseudo-image generation process, the label generation process, and the training data generation process multiple times. The training data generation device, the training data generation method, and the training data generation program implemented therein will be described in more detail below.

[0025] Figure 1 is a block diagram showing the configuration of the learning data generation device in the embodiment. Figure 2 is a diagram showing second learning data as an example. Figures 2A and 2B show learning data for left-right position changes, Figures 2C and 2D show learning data for up-down position changes, Figures 2E and 2F show learning data for diagonal position changes, and Figures 2G and 2H show learning data for scaling changes. Figure 3 is a diagram showing first pseudo-images as an example. Figure 3 shows four different first pseudo-images. Figure 4 is a diagram showing optical flow as an example. Figure 4A shows the optical flow in the case of left-right position changes, Figure 4B shows the optical flow in the case of up-down position changes, Figure 4C shows the optical flow in the case of diagonal position changes, and Figure 4D shows the optical flow in the case of scaling changes. Figure 5 is a diagram showing a change amount map and its complementary change amount map that visualize the amount of change as an example. Figure 5A shows the change amount map for horizontal position changes, and Figure 5B shows the interpolated change amount map obtained by supplementing the change amount map shown in Figure 5A. Figure 5C shows the change amount map for vertical position changes, and Figure 5D shows the interpolated change amount map obtained by supplementing the change amount map shown in Figure 5C. Figure 5E shows the change amount map for diagonal position changes, and Figure 5F shows the interpolated change amount map obtained by supplementing the change amount map shown in Figure 5E. Figure 5G shows the change amount map for scaling changes, and Figure 5H shows the change amount map shown in Figure 5G. Note that interpolation is not performed in the case of scaling changes. Figure 6 shows a first pseudo-image and its heat map as an example. Figure 6A shows the first pseudo-image, and Figure 6B shows its heat map.

[0026] The learning data generation device S in this embodiment includes, for example, a control processing unit 1, an input unit 2, an output unit 3, an interface unit (IF unit) 4, and a storage unit 5, as shown in Figure 1.

[0027] The input unit 2 is connected to the control processing unit 1 and is a device that inputs various commands, such as a command to instruct the start of training data generation, and various data necessary for operating the training data generation device S, such as a training dataset for the first model and a training dataset for the second model, to the training data generation device S. For example, it may be a plurality of input switches assigned to predetermined functions, a keyboard, a mouse, etc. The output unit 3 is connected to the control processing unit 1 and is a device that outputs commands and data input from the input unit 2, and training data generated by the training data generation device S, etc., according to the control of the control processing unit 1. For example, it may be a display device such as a CRT display, LCD (liquid crystal display device), or organic EL display, or a printing device such as a printer, etc.

[0028] The input unit 2 and output unit 3 may be configured as touch panels. In this configuration, the input unit 2 is a position input device that detects and inputs the operating position, such as a resistive or capacitive touchscreen, and the output unit 3 is a display device. In this touch panel, a position input device is provided on the display surface of the display device, and one or more candidate input contents that can be input to the display device are displayed. When the user touches the display position that displays the input content they want to input, the position input device detects that position, and the display content displayed at the detected position is input to the learning data generation device S as the user's operation input. With such a touch panel, the user can easily understand the input operation intuitively, thus providing a user-friendly learning data generation device S.

[0029] The IF unit 4 is connected to the control processing unit 1 and, in accordance with the control of the control processing unit 1, is a circuit that inputs and outputs data to and from external devices, for example. Examples include an RS-232C serial communication interface circuit, an interface circuit using the Bluetooth® standard, and an interface circuit using the USB standard. Alternatively, the IF unit 4 may be a communication interface circuit that sends and receives communication signals to and from external devices, such as a data communication card or a communication interface circuit conforming to the IEEE 802.11 standard.

[0030] The memory unit 5 is connected to the control processing unit 1 and is a circuit that stores various predetermined programs and various predetermined data in accordance with the control of the control processing unit 1.

[0031] The various predetermined programs mentioned above include, for example, a control processing program, which includes, for example, a control program, a first model generation program, a second model generation program, a first pseudo-image generation program, a second pseudo-image generation program, a label generation program, a training data generation program, and a training dataset generation program. The control program controls each of the parts 2 to 5 of the training data generation device S according to the function of each part. The first model generation program is a program that performs machine learning on a first model that generates a first pseudo-image based on a first image containing a predetermined object, by using a training dataset for the first model that comprises a plurality of first images and class labels associated with each of the plurality of first images. The second model generation program is a program that generates a plurality of second models, each with a function to geometrically change the object, in which the first model is given the function to geometrically change the object, with each model having a different method of change. The first pseudo-image generation program is a program that executes a first pseudo-image generation process that generates a first pseudo-image from a first image using the first model. The second pseudo-image generation program is a program that executes a second pseudo-image generation process to generate a plurality of second pseudo-images from the first pseudo-image using each of the plurality of second models. The label generation program is a program that executes a label generation process to determine the amount of change between each of the plurality of second pseudo-images and the first pseudo-image, and to generate a label for the first pseudo-image based on the amount of change determined for each of the plurality of second pseudo-images. The training data generation program is a program that executes a training data generation process to generate training data by associating the first pseudo-image with the label. The training dataset generation program is a program that generates multiple training data as a training dataset by repeating the first pseudo-image generation process, the second pseudo-image generation process, the label generation process and the training data generation process multiple times.

[0032] The aforementioned various predetermined data include, for example, a training dataset for the first model, a training dataset for the second model, and training datasets (generated training dataset, third training dataset) of the training data (generated training data, third training data) generated by the training data generation device S, which are necessary for executing each of these programs.

[0033] Such a storage unit 5 may include, for example, a non-volatile memory element such as ROM (Read Only Memory) or a rewritable non-volatile memory element such as EEPROM (Electrically Erasable Programmable Read Only Memory). Furthermore, the storage unit 5 includes RAM (Random Access Memory) which serves as the working memory of the control processing unit 1, storing data generated during the execution of the predetermined program. The storage unit 5 may also be configured to include a hard disk drive with a relatively large storage capacity.

[0034] The memory unit 5 functionally comprises a first learning data storage unit 51, a second learning data storage unit 52, and a generation learning data storage unit 53.

[0035] The first training data storage unit 51 stores a first model training dataset comprising multiple sets of first training data. The first model training dataset is used to generate a first model by machine learning, which generates a first pseudo-image based on a first image containing a predetermined object. For this reason, the first training data comprises a first image containing the predetermined object and a class label (training data) associated with the first image. Thus, the first model training dataset comprises multiple first images and a class label associated with each of the multiple first images. The object may be any object, for example, an object to be detected. In one example, if the object to be detected is a cat, the first image is an image that shows a cat, and the class label is cat. An image with a class label is an image in which the object to be detected is contained somewhere within the image. For this reason, it is sufficient to have multiple images with class labels as the first model training dataset, so the first training data implicitly contains class labels (training data), and therefore it is not necessarily required to explicitly contain class labels (training data). Since the size of the cat's position that can be generated by the first model after machine learning is controlled according to the cat's position and size in the first image, it is preferable that the first image of the first training data be an image of various types depending on the detection range of the target to be detected, and from the viewpoint of generating many variations of the first pseudo-image with the first model, it is preferable that the image is one in which the cat is in various positions and various poses (postures).

[0036] The second learning data storage unit 52 stores a second model learning dataset comprising multiple sets of second learning data. The second model learning dataset is used to generate a second model by machine learning that adds the function of geometrically changing the object to the first model. The geometric changes include at least two of the following: a first change that changes the position of the object in the left-right direction, a second change that changes the position of the object in the up-down direction, a third change that changes the position of the object diagonally, and a fourth change that either enlarges or reduces the object. In this embodiment, all four are included. Therefore, four second A, second B, second C, and second D model learning datasets are prepared and stored in the second learning data storage unit 52.

[0037] The training dataset for the 2A model is used to generate a second model (2A model) by machine learning, which is the first model with the added function of changing the position of the object in the left-right direction (left-right position change function). For this reason, the second training data (2A training data) of the training dataset for the 2A model includes, for example, an image of a cat shown in Figure 2A with the cat positioned slightly to the left of the center (left-aligned image) and an image of a cat shown in Figure 2B with the cat positioned slightly to the right of the center (right-aligned image). Such 2A training data is generated by overlaying an image of a cat (cat image) onto an image with a solid color background (e.g., black). The left-aligned image can be generated by randomly setting the overlay position to the left of the center, and the right-aligned image can be generated by randomly setting the overlay position to the right of the center. The cat images in the left-aligned and right-aligned images of the 2A training data are identical. Multiple such 2A training data sets are generated, each being different from the others, to create the training dataset for the 2A model.

[0038] The training dataset for the 2B model is used to generate a second model (2B model) by machine learning, which is the first model with the added function of changing the vertical position of the object (vertical position change function). For this reason, the second training data (2B training data) of the training dataset for the 2B model includes, for example, an image of the cat shown in Figure 2C where the cat is positioned above the center (upper-angled image) and an image of the cat shown in Figure 2D where the cat is positioned below the center (lower-angled image). Such 2B training data is generated by overlaying the cat image onto the background image. The upper-angled image can be generated by randomly setting the overlay position above the center, and the lower-angled image can be generated by randomly setting the overlay position below the center. The cat images in the upper-angled image and lower-angled image in the 2B training data are identical. Multiple such 2B training data sets are generated so that they are different from each other, and the training dataset for the 2B model is generated.

[0039] The training dataset for the second C model is used to generate a second model (second C model) by machine learning, which is the first model with the added function of changing the position of the object diagonally (diagonal position change function). For this purpose, the second training data (second C training data) of the second C training dataset comprises, for example, one diagonal image and the other diagonal image, and each of the one diagonal image and the other diagonal image is an image in which, when the one diagonal image and the other diagonal image are placed side by side, the line segment connecting the central position of the cat image in the one diagonal image (one central position) and the central position of the cat image in the other diagonal image (other central position) intersects with a line segment along the direction of placement. Such second C training data can be generated by superimposing the cat image onto the background image, and when superimposing, the angle of intersection is randomly generated, and the positions where each cat image in the one diagonal image and the other diagonal image are superimposed are randomly generated on the line segment. Each cat image in the one-sided oblique image and the other-sided oblique image in the 2C training data are identical. The one-sided oblique image shown in Figure 2E is an image in which the cat image is superimposed diagonally to the lower left of the center of the background image, and the other-sided oblique image shown in Figure 2F is an image in which the cat image is superimposed diagonally to the upper right of the aforementioned center of the background image. Multiple such 2C training data sets are generated so that they are all different from each other, and the training dataset for the 2C model is generated.

[0040] The training dataset for the 2D model is used to generate a second model (2D model) by machine learning, which is the first model with the added function of either enlarging or shrinking the object (enlargement / reduction function). For this reason, the second training data (2D training data) of the training dataset for the 2D model comprises, for example, one enlarged / shrunk image and the other enlarged / shrunk image, where the size of the cat image in the one enlarged / shrunk image and the size of the cat image in the other enlarged / shrunk image are different from each other, as shown in Figures 2G and 2H, for example. Such 2D training data can be generated by overlaying the cat image onto the background image, and by randomly generating the sizes of each image so that the size of the cat image in the one enlarged / shrunk image and the size of the cat image in the other enlarged / shrunk image are different from each other during the overlaying process. When resizing, for example, the original image is enlarged by pixel interpolation, and for example, the original image is reduced by pixel decimation. Both the cat image in the first enlarged image and the cat image in the second enlarged image may be resized, or one of them may be resized. The cat images in the first enlarged image and the second enlarged image in the 2D training data differ in size, but the subject and angle are the same. The first enlarged image shown in Figure 2G is an image in which the size of the cat image is smaller than the size of the cat image in the second enlarged image shown in Figure 2H. In other words, the second enlarged image shown in Figure 2H is an image in which the size of the cat image is larger than the size of the cat image in the first enlarged image shown in Figure 2G. Multiple such 2D training data sets are generated so that they are different from each other, and the training dataset for the 2D model is generated.

[0041] The generative learning data storage unit 53 stores a learning dataset (generative learning dataset, learning dataset for the third model) which comprises multiple learning data (generative learning data, third learning data) generated by the learning data generation device S. The generative learning data comprises the first pseudo-image, which is generated as described below, and the label of the first pseudo-image associated with the first pseudo-image.

[0042] The control processing unit 1 is a circuit for generating the generated learning dataset by controlling each of the parts 2 to 5 of the learning data generation device S according to the function of each part. The control processing unit 1 is configured, for example, with a CPU (Central Processing Unit) and its peripheral circuits. When the control processing program is executed, the control unit 11, the first model generation unit 12, the second model generation unit 13, the first pseudo-image generation unit 14, the second pseudo-image generation unit 15, the label generation unit 16, the learning data generation unit 17, and the learning dataset generation unit 18 are functionally configured in the control processing unit 1.

[0043] The control unit 11 controls each of the parts 2 to 5 of the learning data generation device S according to the function of each part, and is in charge of the overall control of the learning data generation device S.

[0044] The first model generation unit 12 performs machine learning on a first model that generates a first pseudo-image based on a first image containing a predetermined object, by using a first model training dataset comprising multiple first training data. The first model has a latent space in its architecture, and by manipulating the latent space, it is possible to change the position and size of the object to be detected in the image. For example, the known StyleGAN or its improvement, StyleGAN2, can be used as the first model. In StyleGAN and StyleGAN2, latent variables of the latent space are acquired through machine learning. In this embodiment, the first model generation unit 12 performs machine learning on an untrained StyleGAN2 using the first model training dataset stored in the first training data storage unit 51 of the storage unit 5, thereby generating a trained StyleGAN2 as the first model, and stores this generated first model (the trained StyleGAN2 in this example) in the storage unit 5. Figure 3 shows four examples of first pseudo-images generated by the first model generated in this way.

[0045] The second model generation unit 13 generates multiple second models, each having a function to geometrically change the object, by adding this function to the first model, and each second model has a different method of change. As described above, the geometric methods of change include at least two of the four first to fourth methods of change, and in this embodiment, all four are included. More specifically, the second model generation unit 13 first duplicates the first model (in this example, the machine-trained StyleGAN2) that has been machine-trained by the first model generation unit 13 into four copies and stores them in the storage unit 5. Subsequently, the second model generation unit 13 uses the training dataset for the second A model stored in the second training data storage unit 52 of the storage unit 5 to machine-train one of the four first models, thereby generating a second A model to which the left-right position change function has been added to the first model, and stores this generated second A model (in this example, the machine-trained StyleGAN2 with the left-right position change function) in the storage unit 5. Subsequently, the second model generation unit 13... The second model generation unit 13 uses the training dataset for the second B model stored in the second training data storage unit 52 of the storage unit 5 to machine-learn one of the remaining three first models, thereby generating a second B model to which the vertical position change function has been added to the first model, and stores this generated second B model (in this example, a machine-learned StyleGAN2 to which the vertical position change function has been added) in the storage unit 5. Next, the second model generation unit 13 uses the training dataset for the second C model stored in the second training data storage unit 52 of the storage unit 5 to machine-learn one of the remaining two first models, thereby generating a second C model to which the diagonal position change function has been added to the first model, and stores this generated second C model (in this example, a machine-learned StyleGAN2 to which the diagonal position change function has been added) in the storage unit 5. Then, the second model generation unit 13... Using the training dataset for the second D model stored in the second training data storage unit 52 of the storage unit 5, the remaining one first model is subjected to machine learning to generate a second D model to which the scaling function has been added to the first model, and this generated second D model (in this example, a trained StyleGAN2 with the scaling function added) is stored in the storage unit 5.Such a 2A model can change the detection target within the range of the position and size of the detection target in each of the training datasets for the 2A model used in machine learning. The same applies to the 2B to 2D models.

[0046] The first pseudo-image generation unit 14 performs a first pseudo-image generation process that generates a first pseudo-image from the first image using the first model. In this embodiment, the first pseudo-image generation unit 14 inputs the first image to a machine-learned StyleGAN2, generates a first pseudo-image as its output, and stores the generated first pseudo-image in the storage unit 5.

[0047] The second pseudo-image generation unit 15 performs a second pseudo-image generation process that generates a plurality of second pseudo-images from the first pseudo-image using each of the plurality of second models. In this embodiment, the second pseudo-image generation unit 15 first inputs the first pseudo-image generated by the first pseudo-image generation unit 14 to the second A model (in this example, a machine-learned StyleGAN2 with the left-right position change function added), thereby generating a second pseudo-image (second A pseudo-image) as its output, and stores this generated second A pseudo-image in the storage unit 5 in association with the first pseudo-image. Subsequently, the second pseudo-image generation unit 15 inputs the first pseudo-image to the second B model (in this example, a machine-learned StyleGAN2 with the up-down position change function added), thereby generating a second pseudo-image (second B pseudo-image) as its output, and stores this generated second B pseudo-image in the storage unit 5 in association with the first pseudo-image. Next, the second pseudo-image generation unit 15 inputs the first pseudo-image into the second C model (in this example, a machine-learned StyleGAN2 with the diagonal position change function added), generating a second pseudo-image (second C pseudo-image) as its output, and stores this generated second C pseudo-image in the storage unit 5 in association with the first pseudo-image. Then, the second pseudo-image generation unit 15 inputs the first pseudo-image into the second D model (in this example, a machine-learned StyleGAN2 with the scaling change function added), generating a second pseudo-image (second D pseudo-image) as its output, and stores this generated second D pseudo-image in the storage unit 5 in association with the first pseudo-image. Therefore, one first pseudo-image generated by the first pseudo-image generation unit 14 is associated with four second A to second D pseudo-images.

[0048] Furthermore, a second pseudo-image that shows a large amount of change relative to the first pseudo-image (a second pseudo-image with a large amount of change) may be discarded (deleted) (a second pseudo-image may be generated within a predetermined range of change relative to the first pseudo-image). For example, a threshold is set according to the manner of change, and the second pseudo-image is generated by changing the latent variable related to the manner of change in the latent space's latent variables within the range of the threshold. This generates a second pseudo-image within a predetermined range of change relative to the first pseudo-image.

[0049] The label generation unit 16 calculates the amount of change between each of the plurality of second pseudo-images and the first pseudo-image, and performs a label generation process to generate a label for the first pseudo-image based on the amount of change calculated for each of the plurality of second pseudo-images. The second model geometrically changes the object (in one example, the object to be detected), so by calculating the amount of change between the second pseudo-image and the first pseudo-image, the position of the object in the first pseudo-image can be estimated, and a label can be generated. More specifically, in this embodiment, as part of the label generation process, the label generation unit 16 first calculates the amount of change between each of the plurality of second pseudo-images and the first pseudo-image to generate change amount data, then for each of the change amount data calculated for each of the plurality of second pseudo-images, it fills in any missing change amount data to generate complementary change amount data, and then generates the label based on the complementary change amount data generated for each of the change amount data. The change between the first pseudo-image and the second pseudo-image is represented by optical flow.

[0050] More specifically, the label generation unit 16 first obtains the optical flow of the second A pseudo-image relative to the first pseudo-image as change amount data. That is, the label generation unit 16 divides the first pseudo-image into a mesh-like (grid-like) area of ​​a predetermined size, and for each area, estimates (searches) the area of ​​the second pseudo-image corresponding to the area of ​​the first pseudo-image using, for example, the Lucas-Kanade method or the Horn-Schunk method, and obtains the displacement vector from the area of ​​the first pseudo-image to the estimated area of ​​the second pseudo-image as the optical flow of that area. An example of this is shown in Figure 4. In Figure 4, the optical flows of each area are represented by arrows superimposed on the first pseudo-image. The direction of the arrow represents the direction of movement of the optical flow, and the length of the arrow represents the magnitude of the optical flow.

[0051] Next, the label generation unit 16 determines whether the size of the optical flow in each region is equal to or greater than a predetermined threshold (first determination threshold), and extracts regions with an optical flow size equal to or greater than the first determination threshold. This extracts regions with a change amount corresponding to the first determination threshold. An example of a change amount map, in which regions with a change amount corresponding to the first determination threshold are represented by white circles (○), is shown in Figures 5A, 5C, and 5E. Here, when extracting regions with large change amounts, the edges of the detection target may change significantly, but the interior of the detection target may change so little that it is not extracted. For this reason, missing regions are filled in according to the way the change occurs. In the change amount map for left-right position changes, if there is a region that is not extracted as a region with a change amount corresponding to the first determination threshold between two regions with a change amount corresponding to the first determination threshold in the left-right direction, this is treated as a missing region and filled in, and this missing region is changed to a region with a change amount corresponding to the first determination threshold. As a result, for example, the change amount map in the case of left-right position change shown in Figure 5A becomes the interpolated change amount map shown in Figure 5B after interpolation. In the change amount map for up-down position change, if there is a region in the up-down direction between two regions that have a change amount corresponding to the first judgment threshold that has not been extracted as a region with a change amount corresponding to the first judgment threshold, this region is treated as the missing region and interpolated, and this missing region is changed to a region with a change amount corresponding to the first judgment threshold. As a result, for example, the change amount map in the case of up-down position change shown in Figure 5C becomes the interpolated change amount map shown in Figure 5D after interpolation. In the change amount map for diagonal position change, if there is a region in the diagonal direction between two regions that have a change amount corresponding to the judgment threshold that has not been extracted as a region with a change amount corresponding to the first judgment threshold, this region is treated as the missing region and interpolated, and this missing region is changed to a region with a change amount corresponding to the first judgment threshold. As a result, for example, the change amount map in the case of left-right position change shown in Figure 5E becomes the interpolated change amount map shown in Figure 5F after interpolation.

[0052] In the case of scaling changes, regions with a large overlap of optical flow displacement vectors are extracted as the central region of the detection target. That is, the label generation unit 16 determines for each region whether the number of displacement vectors overlapping that region is equal to or greater than a predetermined threshold (second judgment threshold), and thereby extracts regions with a number of optical flows equal to or greater than the second judgment threshold. In Figures 5G and 5H, the degree of overlap of displacement vectors is represented as white lines, forming a change amount map. No interpolation is performed in the case of scaling changes.

[0053] The label generation unit 16 then counts the number of times each interpolated change amount data obtained for each of the plurality of second pseudo-images in each region has been extracted as a region with a change amount corresponding to the first judgment threshold, and generates a heatmap as the label for the first pseudo-image based on this count. The more times a region has been extracted as having a change amount corresponding to the first judgment threshold, the higher the probability that the object (in one example, the detection target) is located in that region. By using the heatmap of the number of times a region has been extracted as having a change amount corresponding to the first judgment threshold as a label, the position of the object in the first pseudo-image can be probabilistically indicated. An example of such a heatmap is shown in Figures 6B, 6D, 6F, and 6H. Figure 6B is a heatmap of the first pseudo-image shown in Figure 6A, Figure 6D is a heatmap of the first pseudo-image shown in Figure 6C, Figure 6F is a heatmap of the first pseudo-image shown in Figure 6E, and Figure 6H is a heatmap of the first pseudo-image shown in Figure 6G. Each of these heatmaps is displayed overlaid on each of the first pseudo-images. In each of these heatmaps, regions extracted three times as having a change amount corresponding to the first judgment threshold are represented by open circles (○), regions extracted twice as having a change amount corresponding to the first judgment threshold are represented by open diamonds (◇), regions extracted once as having a change amount corresponding to the first judgment threshold are represented by open triangles (△), and regions extracted zero times as having a change amount corresponding to the first judgment threshold (regions not extracted as having a change amount corresponding to the first judgment threshold) are represented by open rectangles (□). In each of these heatmaps, regions with an optical flow number equal to or greater than the second judgment threshold are represented as the central region of the detection target by a white pentagon.

[0054] As mentioned above, this heatmap may be used as the label for the first pseudo-image. For example, the area with 3 occurrences may be labeled as the area of ​​the cat, or pixels in areas with 2 or more occurrences may be labeled as the pixels of the cat.

[0055] The learning data generation unit 17 performs a learning data generation process that generates learning data by associating the first pseudo-image with the label. More specifically, the learning data generation unit 17 associates the first pseudo-image generated by the first pseudo-image generation unit 14 with the label of the first pseudo-image generated by the label generation unit 16 based on the first pseudo-image and a plurality of second pseudo-images generated from the first pseudo-image by the second pseudo-image generation unit 15, and stores it in the storage unit 5's learning data storage unit 53 as one of the learning data generated in the learning data set.

[0056] The training dataset generation unit 18 generates multiple training data as training datasets by repeating the first pseudo-image generation process, the second pseudo-image generation process, the label generation process, and the training data generation process multiple times.

[0057] These control processing unit 1, input unit 2, output unit 3, IF unit 4, and storage unit 5 can be configured using, for example, a desktop or notebook computer.

[0058] Next, the operation of this embodiment will be described. Figure 7 is a flowchart showing the operation of the learning data generation device.

[0059] When the power is turned on, the learning data generation device S with this configuration performs the initialization of each necessary part and starts operating. The control processing unit 1 is functionally configured with a control unit 11, a first model generation unit 12, a second model generation unit 13, a first pseudo-image generation unit 14, a second pseudo-image generation unit 15, a label generation unit 16, a learning data generation unit 17, and a learning dataset generation unit 18, through the execution of its control processing program.

[0060] Before the generation of the generative training dataset, the first and second training datasets are stored in the first and second training data storage units 51 and 52, respectively, in the storage unit 5. For example, the first and second training datasets are input from the input unit 2 and stored in the storage unit 5. Alternatively, the first and second training datasets may be input via the IF unit 4 from a storage medium (e.g., a USB memory stick or SD card) on which they are stored and stored in the storage unit 5, or input via the drive device and IF unit 4 from a recording medium (e.g., a CD-R or DVD-R) on which they are recorded and stored in the storage unit 5, or input via the communication network and IF unit 4 from a management server device that manages them and stored in the storage unit 5. Alternatively, the first training dataset may be input and stored, and the second training dataset may be generated from the training data of this first training dataset and stored.

[0061] In the generation of the generative learning dataset, as shown in Figure 7, the learning data generation device S first generates a first model using the first model generation unit 12 of the control processing unit 1, and stores this generated first model in the storage unit 5 (S1, step 7).

[0062] Next, the learning data generation device S generates a plurality of second models using the second model generation unit 13 of the control processing unit 1, and stores these generated plurality of second models in the storage unit 5 (S2, first step). In this embodiment, four second A to second D models are generated and stored.

[0063] Next, the learning data generation device S generates a first pseudo-image from the first image using the first model by the first pseudo-image generation unit 14 of the control processing unit 1, and stores the generated first pseudo-image in the storage unit 5 (S3, first pseudo-image generation process, second step).

[0064] Next, the learning data generation device S, using the second pseudo-image generation unit 15 of the control processing unit 1, generates a plurality of second pseudo-images from the first pseudo-image generated by the first pseudo-image generation unit 14 in process S3, using each of the plurality of second models generated by the second model generation unit 13 in process S2, and stores these generated plurality of second pseudo-images in the storage unit 5 in association with the first pseudo-image (S4, second pseudo-image generation process, third step). In this embodiment, four second A to second D pseudo-images are generated and stored for each of the four second A to second D models.

[0065] Next, the learning data generation device S, using the label generation unit 16 of the control processing unit 1, calculates the amount of change between each of the plurality of second pseudo-images and the first pseudo-image and generates change amount data, and stores this generated change amount data in the storage unit 5 (S5, change amount calculation processing of the label generation process, 41st step of the 4th step). In this embodiment, the optical flow of each region into which the first pseudo-image is divided is obtained as change amount data.

[0066] Next, the learning data generation device S, using the label generation unit 16 of the control processing unit 1, generates complementary change amount data for each of the change amount data obtained in S5 for each of the plurality of second pseudo-images by filling in the missing change amount data, and stores this generated complementary change amount data in the storage unit 5 (S6, complementary processing of the label generation process, step 42 of the fourth process). In this embodiment, a change amount map is generated based on the optical flow of each region into which the first pseudo-image is divided, and a complementary change amount map is generated by interpolating this generated change amount map according to the way the change occurs.

[0067] Next, the learning data generation device S generates labels for each of the change amount data based on the complementary change amount data generated in process S6 by the label generation unit 16 of the control processing unit 1 (S7, label processing of the label generation process, 43rd step of the 4th process).

[0068] Next, the learning data generation device S generates learning data (generated learning data) by associating the first pseudo-image generated by the first pseudo-image generation unit 14 in process S3 with the label generated by the label generation unit 16 in process S7, using the learning data generation unit 17 of the control processing unit 1. This generated generated learning data is then stored in the generated learning data storage unit 53 of the storage unit 5 (S8, learning data generation process, fifth step).

[0069] Next, the learning data generation device S, using the learning data set generation unit 18 of the control processing unit 1, determines whether the process has ended (S9, sixth step). If the result of this determination is that the process has ended (Yes), the learning data generation device S then executes process S10. On the other hand, if the result of the determination is that the process has not ended (No), the learning data generation device S returns to process S3. In this way, the first pseudo-image generation process, the second pseudo-image generation process, the label generation process, and the learning data generation process (the second to fifth steps) are repeated. In the determination, for example, if the input unit 2 receives a command instructing the end of the process, or if the first pseudo-image generation process, the second pseudo-image generation process, the label generation process, and the learning data generation process (the second to fifth steps) have been repeated a predetermined number of times, the process is determined to be finished.

[0070] In the process S10 described above, the learning data generation device S outputs the generated learning dataset stored in the generated learning data storage unit 53 of the storage unit 5 to the output unit 3 via the control unit 11 of the control processing unit 1, and terminates this process. If necessary, the generated learning dataset may be output to an external device via the IF unit 6.

[0071] Processes S1 and S3 are processes for generating images of the generative training data, and processes S2 and S4 through S7 are processes for generating labels for the generative training data (labels for the images).

[0072] As described above, the learning data generation device S in the embodiment, as well as the learning data generation method and learning data generation program implemented therein, can generate labels for the first pseudo-image generated by the first model by using the second model generated based on the first model. Therefore, annotation is unnecessary, and the amount of manual annotation required can be further reduced.

[0073] The above-described training data generation device S, training data generation method, and training data generation program can generate labels with greater accuracy because they fill in missing change data. In particular, because they fill in according to the way the data changes, the above-described training data generation device S, training data generation method, and training data generation program can fill in more appropriately and generate labels with greater accuracy.

[0074] The above-described learning data generation device S, learning data generation method, and learning data generation program can generate a first model by machine learning. Therefore, the above-described learning data generation device S, learning data generation method, and learning data generation program can generate a first model while taking into account features that humans might not have noticed.

[0075] According to this embodiment, a learning data generation device S, a learning data generation method, and a learning data generation program can be provided that include at least two of the first to fourth ways of change.

[0076] According to this embodiment, a learning data generation device S, a learning data generation method, and a learning data generation program can be provided that represent the amount of change between the first pseudo-image and the second pseudo-image using optical flow.

[0077] To illustrate the present invention, the embodiments have been adequately and fully described above with reference to the drawings. However, those skilled in the art should recognize that it is easy to modify and / or improve upon the embodiments described above. Therefore, unless such modifications or improvements implemented by those skilled in the art fall outside the scope of the claims, such modifications or improvements shall be considered to be included within the scope of the claims. [Explanation of Symbols]

[0078] S Learning Data Generator 1 Control Processing Unit 5 Storage section 11 Control Unit 12. First Model Generation Unit 13. Second Model Generation Unit 14 First pseudo image generation section 15 Second pseudo image generation section 16 Label generation unit 17. Training Data Generation Unit 18. Training Dataset Generation Unit 51 First Learning Data Storage Unit 52 Second Learning Data Storage Unit 53 Generative learning data storage unit

Claims

1. A first step involves generating multiple second models, each with a different method of transformation, which are provided with a function to geometrically change the object in a first model that generates a first pseudo-image based on a first image containing a predetermined object, and A second step involves generating a first pseudo-image from a first image using the first model described above, A third step of generating a plurality of second pseudo-images from the first pseudo-image using each of the plurality of second models, A fourth step involves determining the amount of change between each of the plurality of second pseudo-images and the first pseudo-image, and generating a label for the first pseudo-image based on the amount of change determined for each of the plurality of second pseudo-images. A fifth step involves associating the first pseudo-image with the label to generate training data, The method comprises a sixth step of generating multiple training data as a training dataset by repeating the second to fifth steps multiple times. Method for generating training data.

2. The aforementioned fourth step is, A 41st step involves determining the amount of change between each of the plurality of second pseudo-images and the first pseudo-image and generating change amount data, For each of the plurality of second pseudo-images, a 42nd step is to generate complementary change data by filling in the missing change data for each of the change data obtained in the 41st step, The process includes a 43rd step of generating the label for each of the aforementioned change amount data based on the complementary change amount data generated in the 42nd step. The method for generating training data according to claim 1.

3. The first model is STYLEGAN or STYLEGAN2, By using a training dataset for a first model comprising multiple first images and class labels associated with each of the multiple first images, the first model is further subjected to a seventh step of machine learning, which precedes the first step. The method for generating training data according to claim 1.

4. The method of change includes at least two of the following: a first method of change in which the object is moved in the left-right direction; a second method of change in which the object is moved in the up-down direction; a third method of change in which the object is moved in the diagonal direction; and a fourth method of change in which the object is either enlarged or reduced. The first step generates at least two second models by performing at least two of the above steps. The method for generating training data according to claim 1.

5. The change between the first pseudo-image and the second pseudo-image is represented by optical flow. The method for generating training data according to claim 1.

6. On the computer, A first step involves generating multiple second models, each with a different method of transformation, which are provided with a function to geometrically change the object in a first model that generates a first pseudo-image based on a first image containing a predetermined object, and A second step involves generating a first pseudo-image from a first image using the first model described above, A third step of generating a plurality of second pseudo-images from the first pseudo-image using each of the plurality of second models, A fourth step involves determining the amount of change between each of the plurality of second pseudo-images and the first pseudo-image, and generating a label for the first pseudo-image based on the amount of change determined for each of the plurality of second pseudo-images. A fifth step involves associating the first pseudo-image with the label to generate training data, A sixth step involves generating multiple training data as a training dataset by repeating the second to fifth steps described above multiple times, A program for generating training data to run the program.

7. A second model generation unit generates multiple second models, each with a different method of transformation, which are added to a first model that generates a first pseudo-image based on a first image containing a predetermined object, and which have a function to geometrically transform the object. A first pseudo-image generation unit that executes a first pseudo-image generation process to generate a first pseudo-image from a first image using the first model, A second pseudo-image generation unit performs a second pseudo-image generation process that generates a plurality of second pseudo-images from the first pseudo-image using each of the plurality of second models, A label generation unit performs a label generation process that calculates the amount of change between each of the plurality of second pseudo-images and the first pseudo-image, and generates a label for the first pseudo-image based on the amount of change calculated for each of the plurality of second pseudo-images. A learning data generation unit that performs a learning data generation process to generate learning data by associating the first pseudo-image with the label, The system includes a learning dataset generation unit that generates multiple learning data as a learning dataset by repeating the first pseudo-image generation process, the second pseudo-image generation process, the label generation process, and the learning data generation process multiple times. A device for generating training data.

Images