Training data generation device, method, and program, and object detection device, method, and program

Abstract

To provide: a learning data generating apparatus, method and program capable of further reducing the number of man-hours for annotation; and an object detecting apparatus, method and program for machine-learning a machine learning model for detecting a detection target, using the learning data generated thereby.SOLUTION: A learning data generating apparatus of the present invention uses a predetermined machine learning method based on a plurality of real images including a predetermined target object and a plurality of mask images composed of one or more target schematic images schematically representing the target object randomly arranged in a background, thereby generates a plurality of schematic images by excluding a first image region in which the target schematic images in the mask images are arranged to replace the remaining second image region with the real image, and forms a set of the mask images and the schematic images corresponding to the mask images for each of the plurality of mask images to generate a plurality of learning data sets of the mask images and the schematic images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a training data generation device, a training data generation method, and a training data generation program that generate training data to be used in machine learning, as well as an object detection device, an object detection method, and an object detection program that use the training data generated thereby to train a machine learning model that detects a detection target. [Background technology]

[0002] In recent years, machine learning has been researched and developed, and is being applied to various fields. This machine learning requires a relatively large amount of training data to perform the machine learning. In particular, in supervised machine learning, the training data must be accompanied by the supervising data, i.e., supervising data (supervising labels) that indicate whether the training data is correct or not. This requires generating supervising data for each training data (annotation), which results in a significant amount of work. For this reason, there is a demand for reducing the amount of work, and one such technique is disclosed in Non-Patent Document 1, for example.

[0003] The machine learning method disclosed in Non-Patent Document 1 uses a Generative Adversarial Network (GAN) to learn the inter-domain correspondence between two image datasets in different domains, rather than the correspondence between pixels of a pair of images. This machine learning method uses a cyclic structure of transformation and inverse transformation for two image datasets, allowing for machine learning without preparing a large number of image datasets as a training dataset. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] LanIan Liu, et al. “Generative Modeling for SmallData Object Detection”, [online], retrieved January 28, 2020, https: / / arxiv.org / pdf / 1910.017169.pdf Summary of the Invention [Problem to be solved by the invention]

[0005] The machine learning method disclosed in Non-Patent Document 1 does not require a large amount of image data as a training dataset, but this does not mean that it is unnecessary; a small amount of training dataset is required, and annotations corresponding to this are required.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and its purpose is to provide a training data generation device, a training data generation method, and a training data generation program that can further reduce the amount of annotation work, as well as an object detection device, an object detection method, and an object detection program that use the training data generated thereby to train a machine learning model that detects a detection target. [Means for solving the problem]

[0007] After extensive investigation, the inventors have found that the above object can be achieved by the present invention described below. That is, a training data generation device according to one aspect of the present invention uses a predetermined machine learning method based on a plurality of real images including a predetermined object and a plurality of mask images each including one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images to generate a plurality of simulated images in which a remaining second image region in the mask images, excluding a first image region in which the target schematic images are arranged, is replaced with a real image, and a training data set is generated including a plurality of pairs of the mask image and the simulated image by combining, for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image into a single pair.

[0008] Such a training data generation device generates a simulated image based on an actual image and a mask image, and pairs the simulated image with the mask image to generate training data using the mask image as training data. The training data generation device generates a training dataset including a plurality of such training data. Therefore, the training data generation device can further reduce the number of steps required for annotation.

[0009] In another aspect, in the above-described training data generation device, the machine learning method is CycleGAN.

[0010] Such a training data generation device uses CycleGAN as a machine learning method, and can therefore appropriately generate a simulated image based on the actual image and the mask image so that the background portion of the mask image, excluding the target schematic image, becomes the actual image.

[0011] In another aspect, in the above-mentioned training data generation device, the CycleGAN machine learning method includes a detector that identifies and detects a background image from the simulated image excluding the first image region of the object, and an object image from the simulated image that includes only the first image region of the object.

[0012] Such a training data generation device includes a detector that distinguishes and detects a background image and an object image from the simulated image, and therefore can generate simulated images more appropriately.

[0013] In another aspect of the present invention, the object detection device is a detection target, and includes any one of the above-mentioned training data generation devices and a machine learning unit that performs machine learning to create a machine learning model that detects the detection target from an image using a training data set generated by the training data generation device.

[0014] This provides an object detection device that uses a training data set generated by a training data generation device to train a machine learning model for detecting a detection target. Because the object detection device includes the training data generation device, the machine learning unit can train the machine learning model without preparing a separate training data set.

[0015] A training data generation method according to another aspect of the present invention is a computer-executable method that uses a predetermined machine learning method based on a plurality of real images including a predetermined object and a plurality of mask images each including one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images to generate a plurality of simulated images in which the remaining second image areas in the mask images, excluding the first image area in which the target schematic images are arranged, are replaced with real images, and for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image are grouped into one pair, thereby generating a training data set including a plurality of pairs of the mask image and the simulated image.

[0016] A training data generation program according to another aspect of the present invention is a program executed by a computer that uses a predetermined machine learning method based on a plurality of real images including a predetermined object and a plurality of mask images each including one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images to generate a plurality of simulated images in which the remaining second image areas in the mask images, excluding the first image area in which the target schematic images are arranged, are replaced with real images, and for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image are grouped into one pair, thereby generating a training data set including a plurality of pairs of the mask image and the simulated image.

[0017] Such a training data generation method and a training data generation program generate a simulated image based on an actual image and a mask image, and pair the simulated image with the mask image to generate training data using the mask image as training data. The training data generation method and the training data generation program generate a training data set including a plurality of such training data. Therefore, the training data generation method and the training data generation program can further reduce the number of steps for annotation.

[0018] An object detection method according to another aspect of the present invention is a method executed by a computer, comprising: a training data generation step of using a predetermined machine learning method to generate a plurality of simulated images in which a remaining second image area in the mask images, excluding a first image area in which the target schematic images are arranged, is replaced with a real image, based on a plurality of real images including a predetermined object to be detected, and a plurality of mask images each including one or more target schematic images that schematically represent the object and a plurality of mask images each randomly arranged on a monochrome background different from the target schematic images; and a machine learning step of using the training data set generated in the training data generation step to train a machine learning model that detects the object to be detected from an image.

[0019] According to another aspect of the present invention, an object detection program is a computer-executable program that includes: a training data generation process that uses a predetermined machine learning method to generate a plurality of simulated images in which the remaining second image areas in the mask images, excluding the first image area in which the target schematic images are arranged, are replaced with real images, based on a plurality of real images containing a predetermined object to be detected, and a plurality of mask images each consisting of one or more target schematic images that schematically represent the object and randomly arranged on a monochrome background different from the target schematic images; and a machine learning process that uses the training data set generated in the training data generation process to train a machine learning model that detects the object to be detected from an image.

[0020] This provides an object detection method and an object detection program for machine learning a machine learning model that detects a detection target using a training dataset generated by the training data generation method and the training data generation program. The object detection method and the object detection program can machine learn a machine learning model in the machine learning process without preparing a separate training dataset. [Effects of the Invention]

[0021] The training data generation device, training data generation method, and training data generation program according to the present invention can further reduce the number of steps required for annotation. According to the present invention, an object detection device, object detection method, and object detection program can be provided that use the training data set thus generated to train a machine learning model for detecting a detection target. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a block diagram showing a configuration of an object detection device with a learning data generation function according to an embodiment. [Figure 2]1A and 1B are diagrams illustrating real images and detection targets used to generate a training dataset. [Figure 3] FIG. 10 is a diagram showing an example of a target schematic image that is manually generated. [Figure 4] FIG. 4 is a diagram showing an example of a mask image using the target schematic image shown in FIG. 3. [Figure 5] 10A and 10B are diagrams for explaining a method for automatically generating a target schematic image. [Figure 6] FIG. 10 is a diagram showing an example of an automatically generated target schematic image. [Figure 7] FIG. 7 is a diagram showing an example of a mask image using the target schematic image shown in FIG. 6. [Figure 8] FIG. 1 is a diagram for explaining a simulated image generation model used to generate a training dataset. [Figure 9] FIG. 1 is a diagram showing an example of a plurality of real images and a plurality of mask images used to generate a training dataset. [Figure 10] FIG. 2 is a diagram showing an example of supervised learning data generated by the object detection device with the learning data generation function. [Figure 11] FIG. 10 is a diagram illustrating an object detection model used to detect a detection target. [Figure 12] FIG. 10 is a diagram illustrating an example of a detection result using a machine-learned object detection model. [Figure 13] 10 is a flowchart illustrating the operation of the object detection device regarding generation of a training dataset. [Figure 14] 10 is a flowchart illustrating the operation of the object detection device regarding machine learning of an object detection model. [Figure 15] 4 is a flowchart showing the operation of the object detection device regarding detection of a detection target. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, one or more embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the disclosed embodiments. In addition, components with the same reference numerals in each drawing indicate the same components, and their description will be omitted as appropriate. In this specification, when referring to a general term, a reference numeral without a subscript is used, and when referring to an individual component, a reference numeral with a subscript is used.

[0024] A training data generation device according to an embodiment generates a training dataset used in machine learning a machine learning model, particularly a training dataset including teacher data. The training data generation device uses a predetermined machine learning method based on a plurality of real images including a predetermined object and a plurality of mask images each including one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images. The training data generation device generates a plurality of simulated images in which a second image region in the mask image, excluding a first image region in which the target schematic images are arranged, is replaced with a real image. For each of the mask images, the mask image and the corresponding simulated image are paired to generate a training dataset including a plurality of pairs of the mask image and the simulated image. An object detection device according to an embodiment detects a detection target based on an image, and includes the above-described training data generation device, where the detection target is the object, and a machine learning unit that uses the training dataset generated by the training data generation device to train a machine learning model that detects the detection target from an image. Below, we will explain in more detail such a training data generation device, a training data generation method implemented therein, and a training data generation program implemented therein, as well as an object detection device, an object detection method implemented therein, and an object detection program implemented therein, using an object detection device with a training data generation function that integrates the training data generation device and object detection device.

[0025] FIG. 1 is a block diagram showing the configuration of an object detection device with a training data generation function according to an embodiment. FIG. 2 is a diagram illustrating real images and detection targets used to generate a training dataset. FIG. 2A shows an example of a real image, and FIG. 2B is a diagram illustrating the appearance of the detection target reflected in the real image shown in FIG. 2A. FIG. 3 is a diagram illustrating an example of a manually generated target schematic image. FIGS. 3A to 3D show examples of target schematic images of the detection target (object) reflected in an inner region near the center of the real images shown in FIGS. 2A and 2B, and FIGS. 3E to 3G show examples of target schematic images of the detection target (object) reflected in an outer region outside the inner region of the real images shown in FIGS. 2A and 2B. FIG. 4 is a diagram illustrating an example of a mask image using the target schematic image shown in FIG. 3. FIGS. 4A and 4B show examples. FIG. 5 is a diagram illustrating a method for automatically generating target schematic images. FIGS. 5A to 5F show each process. FIG. 6 is a diagram illustrating an example of an automatically generated target schematic image. FIGS. 6A to 6J show examples. FIG. 7 is a diagram showing an example of a mask image using the target schematic image shown in FIG. 6. FIGS. 7A to 7C show examples. FIG. 8 is a diagram for explaining a simulated image generation model used to generate a training dataset. FIG. 9 is a diagram showing an example of multiple real images and multiple mask images used to generate a training dataset. In plan view, the left half of FIG. 9 shows 16 real images, and the right half of FIG. 9 shows 16 mask images. FIG. 10 is a diagram showing an example of supervised training data generated by the object detection device with training data generation function. FIGS. 10A and 10B show examples. FIG. 11 is a diagram for explaining an object detection model used to detect the detection target. FIG. 12 is a diagram showing an example of a detection result using a machine-learned object detection model. FIGS. 12A and 12B are examples, showing an input image on the left side in plan view and a detection result image on the right side.

[0026] The object detection device D with learning data generation function in the embodiment includes, for example, an image acquisition unit 1, a control processing unit 2, an input unit 3, a display unit 4, an interface unit (IF unit) 5, and a memory unit 6, as shown in FIG.

[0027] The image acquisition unit 1 is connected to the control processing unit 2 and is a device that acquires a predetermined image under the control of the control processing unit 2, and the object detection device D executes a process of detecting a predetermined detection target by determining whether or not a predetermined detection target is captured in the acquired predetermined image. The image acquisition unit 1 is, for example, an imaging unit 1 that generates an image in order to detect the detection target substantially in real time, and this imaging unit 1 is, for example, a digital camera that includes an imaging optical system that forms an optical image of the detection target on a predetermined imaging plane, an area image sensor that is arranged so that its light-receiving surface coincides with the imaging plane and converts the optical image of the detection target into an electrical signal, and an image processing unit that processes the output of the area image sensor to generate image data that is data representing the image of the detection target.

[0028] The image acquisition unit 1 is not limited to the imaging unit 1 and may be another device. For example, the image acquisition unit 1 is an interface circuit that inputs and outputs data to and from an external device. The external device may be a storage medium, such as a USB (Universal Serial Bus) memory or an SD card (registered trademark), that stores the predetermined image. Alternatively, the external device may be a drive device that reads data from a storage medium, such as a CD-ROM (Compact Disc Read Only Memory), a CD-R (Compact Disc Recordable), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a DVD-R (Digital Versatile Disc Recordable), that records the predetermined image. The interface circuit serving as the image acquisition unit 1 may be connected to the external device via a wired or wireless connection. Alternatively, the image acquisition unit 1 may be a communication interface circuit that transmits and receives communication signals to and from an external device, and the external device may be a server device that is connected to the communication interface circuit via a network (such as a WAN (Wide Area Network, including a public communication network)) or a LAN (Local Area Network) and manages the predetermined image. In such an image acquisition unit 1, the presence or absence of the detection target can be detected after the predetermined image is generated, and past images can be verified. Here, if the image acquisition unit 1 is an interface circuit or a communication interface circuit, the image acquisition unit 1 may also serve as the IF unit 5 (i.e., the IF unit 5 may be used as the image acquisition unit 1).

[0029] The input unit 3 is connected to the control processing unit 2 and is a device that inputs various commands, such as a command to start generating learning data or a command to start detection, and various data necessary to operate the object detection device D with a learning data generation function, such as actual images and mask images that are the basis for generating learning data, to the object detection device D. The input unit 3 is, for example, a plurality of input switches to which predetermined functions are assigned, a keyboard, a mouse, etc. The display unit 4 is connected to the control processing unit 2 and is a device that displays the commands, data, detection results, etc. input from the input unit 3 under the control of the control processing unit 2. The display unit 4 is, for example, a display device such as a CRT display, an LCD (liquid crystal display), or an organic EL display.

[0030] The input unit 3 and the display unit 4 may be configured as a touch panel. In the case of configuring this touch panel, the input unit 3 is a position input device that detects an operation position and inputs, for example, a resistive film type or a capacitive type. In this touch panel, a position input device is provided on the display surface of the display device, and one or more input content candidates that can be input are displayed on the display device. When a user touches the display position displaying the input content they want to input, the position is detected by the position input device, and the display content displayed at the detected position is input to the object detection device D as the user's operation input content. With such a touch panel, the user can easily intuitively understand the input operation, and therefore an object detection device D that is easy for the user to use is provided.

[0031] The IF unit 5 is connected to the control processing unit 2 and is a circuit that inputs and outputs data to and from, for example, an external device under the control of the control processing unit 2, and is, for example, an interface circuit for RS-232C, which is a serial communication method, an interface circuit using the Bluetooth (registered trademark) standard, an interface circuit using the USB standard, etc. The IF unit 5 may also be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device, such as a data communication card or a communication interface circuit conforming to the IEEE802.11 standard, etc.

[0032] The storage unit 6 is connected to the control processing unit 2 and is a circuit that stores various predetermined programs and various predetermined data under the control of the control processing unit 2. The various predetermined programs include, for example, a control processing program, and the control processing program includes, for example, a control program that controls each of the units 1, 3 to 6 of the object detection device D according to the function of each unit; a first machine learning program that machine-learns a machine learning model (first machine learning model, object detection model) that detects a detection target from an image; a detection program that detects a detection target based on an image acquired by the image acquisition unit 1 by using a machine-learned object detection model that has been machine-learned using the first machine learning program; and a training data generation program that uses a predetermined machine learning method based on a plurality of real images containing a predetermined object and a plurality of mask images formed by randomly arranging one or more target schematic images that schematically represent the object on a solid-color background different from the target schematic images, to generate a plurality of simulated images in which the remaining second image areas in the mask images, excluding the first image area in which the target schematic images are arranged, are replaced with real images, and for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image are paired to generate a training data set including a plurality of pairs of the mask image and the simulated image. The various types of predetermined data include, for example, data necessary for executing each of these programs, such as images and simulated images acquired by the image acquisition unit 1. Such storage unit 6 includes, for example, a ROM (Read Only Memory), which is a nonvolatile storage element, and an EEPROM (Electrically Erasable Programmable Read Only Memory), which is a rewritable nonvolatile storage element. Storage unit 6 also includes a RAM (Random Access Memory), which serves as a working memory for control processing unit 2 and stores data generated during execution of the predetermined programs. Storage unit 6 may also be configured with a hard disk drive having a relatively large storage capacity.

[0033] The control processing unit 2 is a circuit that controls each of the units 1, 3 to 6 of the object detection device D with a learning data generation function according to the function of each unit, detects a predetermined detection target, and generates a predetermined learning data set. The control processing unit 2 is configured, for example, with a CPU (Central Processing Unit) and its peripheral circuits. By executing a control processing program, the control processing unit 2 functionally includes a control unit 21, a first machine learning unit 22, a detection unit 23, and a learning data generation unit 24.

[0034] The control unit 21 controls each of the units 1, 3 to 6 of the object detection device D in accordance with the function of each unit, and is in charge of overall control of the object detection device D.

[0035] The training data generation unit 24 uses a predetermined machine learning method based on multiple real images containing a predetermined object and multiple mask images each consisting of one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images. The training data generation unit 24 generates multiple simulated images in which the remaining second image regions of the mask images, excluding the first image region in which the target schematic images are arranged, are replaced with real images. For each of the multiple mask images, the mask image and the corresponding simulated image are paired to generate a training data set including multiple pairs of the mask image and the simulated image. The machine learning method is CycleGAN, and in this embodiment, the CycleGAN machine learning method includes a structure that identifies a background image from the simulated image excluding the first image region of the object, and an object image from the simulated image that contains only the first image region of the object. Note that the method is not limited to CycleGAN, and any other machine learning method may be used as long as it can generate a mask image and its corresponding simulated image without so-called paired images.

[0036] More specifically, in this embodiment, the training data generation unit 24 functionally includes a second machine learning unit 241 and a generation unit 242.

[0037] The second machine learning unit 241 performs machine learning to create a second machine learning model (simulated image generation model) that generates a simulated image based on a plurality of real images and a plurality of mask images (simulated image learning dataset).

[0038] The object detection device D in this embodiment is a device that detects, for example, a person in a workplace such as a factory as a detection target. Therefore, the actual image is an image including a predetermined object that is the detection target, and is, for example, an image of the workplace viewed from above, more specifically, directly above, as shown in FIG. 2A. Therefore, the imaging unit 1, which is an example of the image acquisition unit 1, is disposed on a rail of a crane or the like so that its optical axis is aligned vertically. Alternatively, for example, the imaging unit 1 is disposed by being suspended from the ceiling of a building so that its optical axis is aligned vertically.

[0039] A target schematic image, which schematically represents the target, is drawn, for example, manually from such a real image. In the example of the real image shown in FIG. 2A, as shown in FIG. 2B, the person to be detected (the target) reflected in the inner area IA near the center of the real image is seen centered on the head, resulting in a target schematic image that mainly resembles the shoulders, arms, and hands, with the head at the center, as shown in, for example, FIGS. 3A to 3D. In the example shown in FIG. 2B, the inner area IA is a rectangular area 25% inward from each side of the real image. On the other hand, as shown in FIG. 2B, the person to be detected (the target) reflected in the outer area OA outside the inner area IA in the real image is seen from the feet to the head, resulting in a target schematic image that resembles the entire person from the feet to the head, as shown in, for example, FIGS. 3E to 3G. Then, by randomly arranging one or more target schematic images generated in this way on a monochrome background different from the target schematic image, multiple mask images, such as those shown in, for example, FIGS. 4A and 4B, are generated. For example, a single-color background is divided into 64 regions (8 × 8) in a two-dimensional matrix, and in the inner 4 × 4 region near the center, target schematic images, for example, as shown in Figures 3A to 3D, are randomly selected and randomly placed, in a number set by random numbers, and in the outer region, target schematic images, for example, as shown in Figures 3E to 3G, are randomly selected and randomly placed, in a number set by random numbers. Here, when these target schematic images are placed, their coordinate positions, rotation angles, and sizes (magnifications for reduction or enlargement) are each set by random numbers for each target schematic image.

[0040] Alternatively, for example, the target schematic image may be generated automatically. First, as shown in FIG. 5A, a direction is set using random numbers, and a body line (a line segment of a predetermined length) is set along the direction. Next, as shown in FIG. 5B, a face is drawn as a predetermined ellipse at a position 20 to 30% above the body line segment. Next, as shown in FIG. 5C, a thickness and length are each set using random numbers, and shoulders are drawn along the line segment below the face using line segments of the thickness and length. Next, as shown in FIG. 5D, a thickness is set using random numbers, and a torso is drawn along the line segment below the shoulders using line segments of the thickness. Next, as shown in FIG. 5E, each thickness, length, and direction are each set using random numbers, and each leg is drawn along the line segment below the torso using line segments of the thickness and length in each direction. In the example shown in FIG. 5E, both legs are overlapping one another. Then, as shown in FIG. 5F, each thickness, length, and direction is set by random numbers, and each arm is drawn in each direction using line segments of each thickness and length from the shoulder. In the example shown in FIG. 5F, both arms are overlapped. This generates the target schematic image, examples of which are shown in FIGS. 6A to 6J. Then, by randomly arranging one or more target schematic images thus generated on a single-color background different from the target schematic image, multiple mask images such as those shown in FIGS. 7A to 7C are generated. Here, when arranging these target schematic images, the number, coordinate position, rotation angle, and size (magnification of reduction or enlargement) are each set by random numbers for each target schematic image.

[0041] In this embodiment, the mask image is a binarized image in which the image region of the target schematic image is white and the background is black.

[0042] The second machine learning model (simulated image generation model) may be the well-known CycleGAN, but in this embodiment, a modified CycleGAN is used, as shown in Figure 8, in which the generator 2411 and the classifier 2412 in CycleGAN further include a detector 2413.

[0043] CycleGAN uses machine learning to find correspondences between two different domains. Let one domain be X and its image data be x, and the other domain be Y and its image data be y.

[0044] The generator 2411 includes a first generator 2411xy that converts an image x in domain X into an image in domain Y, and a second generator 2411yx that converts an image y in domain Y into an image in domain X. The classifier 2412 includes a first classifier 2412y that classifies whether an image yf generated by the first generator 2411xy is an image y in domain Y, and a second classifier 2412x that classifies whether an image xf generated by the second generator 2411yx is an image x in domain X. The CycleGAN has a cyclic structure of transformation and inverse transformation using two such pairs of the first generator 2411xy and first classifier 2412y, and the second generator 2411yx and second classifier 2412x. The first generator 2411xy and the first classifier 2412y constitute a generative adversarial network (GAN), where the first generator 2411xy is machine-learned to deceive the first classifier 2412y, and the first classifier 2412y is machine-learned to detect lies by the first generator 2411xy. Similarly, the second generator 2411yx and the second classifier 2412x constitute a generative adversarial network, where the second generator 2411yx is machine-learned to deceive the second classifier 2412x, and the second classifier 2412x is machine-learned to detect lies by the second generator 2411yx. In this embodiment, domain X is a dataset consisting of a plurality of real images, for example, as shown in the left half of FIG. 9, and domain Y is a dataset consisting of a plurality of mask images, for example, as shown in the right half of FIG. 9. Since CycleGAN uses machine learning to learn correspondences between domains, the actual image and mask image shown in Figure 9 are not paired images.

[0045] The detector 2413 distinguishes and detects a background image, excluding the first image region of the object, from the simulated image and an object image, including only the first image region of the object, from the simulated image. In this embodiment, such a detector 2413 is added to a normal CycleGAN in order to suitably generate a simulated image in which the object (detection target) is placed at the position of the target schematic image in the mask image and an actual image is placed in the remaining region. In each machine learning session of the two sets of the first generator 2411xy and first classifier 2412y and the second generator 2411yx and second classifier 2412x, the detector 2413 performs machine learning simultaneously with the learning timing of the first generator 2411xy and first classifier 2412y. In this case, the background image is generated by the second machine learning unit 241 by replacing pixel values of an image region (corresponding image region) in the input mask image that corresponds to the image region of the target schematic image with 0, in a simulated image generated by inputting an actual image and a mask image to the first generator 2411xy, and leaving pixel values of the remaining image region excluding the corresponding image region unchanged, and is input to the detector 2413. The object image is generated by the second machine learning unit 241 by replacing pixel values of the remaining image region excluding the corresponding image region with 0, in a simulated image generated by inputting an actual image and a mask image to the first generator 2411xy, and leaving pixel values of the corresponding image region unchanged, and is input to the detector 2413.

[0046] If the function (image conversion function) representing the first generator 2411xy is G, the function (image conversion function) representing the second generator 2411yx is F, the function (discrimination function) representing the first classifier 2412y is Dy, and the function (discrimination function) representing the second classifier 2412x is Dx, the relationship between the first generator 2411xy and the first classifier 2412y is expressed by the following equation 1, and the purpose of machine learning of the first generator 2411xy and the first classifier 2412y is to minimize G and maximize Dy in equation 1 as shown in the following equation 2. L in equation 1 GANis generally referred to as Adversarial Loss. Similarly, the objective of the machine learning of the second generator 2411yx and the second classifier 2412x is to minimize F and maximize Dx as shown in the following Equation 3.

[0047]

number

[0048]

number

[0049]

number

[0050] where y~p data (y) is the probability distribution function p data is an expression distributed according to x~p data (x) is the probability distribution function p data where E is the average information content (entropy) (e.g., Ey~p data (y)[logDy(y)]=Σyp data (y)Dy(y)).

[0051] If the above formulas 2 and 3 are used as they are, machine learning will be performed independently. Therefore, in CycleGAN, the following formula 4 is introduced to maintain consistency in the cycle of repeating transformation and inverse transformation. cyc is commonly called Cycle Consistency Loss or Reconstruction Loss.

[0052]

number

[0053] Therefore, the objective of CycleGAN machine learning is to minimize G and F and maximize Dy and Dx in the following equation 5, as shown in the following equation 6.

[0054]

number

[0055]

number

[0056] Here, λ is a weight for controlling the relative importance between the adversarial loss and the circular consistency loss, and is set appropriately in advance.

[0057] The purpose of the machine learning of the modified CycleGAN shown in Figure 8 in this embodiment is to minimize G and maximize Dy in the following equation 7, as shown in the following equation 8, when the function (discrimination function) representing the detector 2413 is Dbo, and to minimize F and maximize Dx, as shown in the following equation 9.

[0058]

number

[0059]

number

[0060]

number

[0061] Returning to FIG. 1 , the generation unit 242 generates multiple pieces of supervised training data and generates a training dataset (a training dataset for detection) by using a machine-learned second machine learning model (a simulated image generation model) that has been machine-learned by the second machine learning unit 241. More specifically, a real image and a mask image are input to the machine-learned simulated image generation model, and the mask image and a simulated image output from the machine-learned simulated image generation model are paired as supervised training data. The mask image serves as the training data, and the simulated image serves as the training data. This process is performed for each of the multiple mask images, and a detection training dataset including multiple pairs of the mask image and the simulated image is generated. Examples of supervised training data are shown in FIGS. 10A and 10B . In FIGS. 10A and 10B , a mask image serving as the training data is shown on the left side in a planar view, and a simulated image serving as the training data is shown on the right side. Note that, since the background image is fixed once the installation position of the object detection device D is determined, the actual image input to the machine-learned simulated image generation model when generating a detection training dataset may be one for each of the multiple mask images, but of course, multiple images may also be input. Also, although the mask images themselves are used as training data, the position coordinates of the target schematic image may be detected from the mask image, and the position coordinates may be used as training data.

[0062] The first machine learning unit 22 performs machine learning on a first machine learning model (object detection model) that detects a detection target from an image, using a training dataset (detection training dataset) generated by the training data generation unit 24. The object detection model may be an image detection method that requires an annotated dataset, such as a convolution neural network (CNN) or a region-based convolution neural network (R-CNN), or may be a rule-based method other than deep learning. In this embodiment, however, the well-known Mask R-CNN shown in FIG. 11 is used. Mask R-CNN is a model that detects objects by instance segmentation, which classifies each pixel of an image into object classes and classifies them as to whether they are the same object. The Mask R-CNN generally includes a backbone network 221 that extracts features of an input image, a region proposal network (RPN) 222 that selects candidate object regions, and a head network 223 that extracts the region candidates obtained by the RPN 222 from the output of the backbone network 221 and performs class identification and region extraction (object detection). In this embodiment, the first machine learning unit 22 performs machine learning on Mask R-CNN as an object detection model using a training dataset generated by the training data generation unit 24, by fine tuning, which is a method of building a new model by reusing part of an existing model. More specifically, the first machine learning unit 22 replaces the head network 223 of Mask R-CNN, which has been machine-trained using a Microsoft Common Object in Context (Microsoft is a registered trademark) dataset, with a head network 223 in which only the head network 223 has been initialized, and fine-tunes the head network 223 using the detection training dataset generated by the training data generation unit 24, thereby generating a machine-trained object detection model that has been machine-trained using the detection training dataset generated by the training data generation unit 24.

[0063] The detection unit 23 detects a detection target based on the image acquired by the image acquisition unit 1 by using a machine-learned object detection model that has been machine-learned by the first machine learning unit 22. The detection result of the detection unit 23 is displayed on the display unit 4. An example of the detection result is shown in FIGS. 12A and 12B. When the image shown on the left side of FIG. 12A is input to the machine-learned object detection model, the image shown on the right side of FIG. 12A is output and displayed on the display unit 4. When the image shown on the left side of FIG. 12B is input to the machine-learned object detection model, the image shown on the right side of FIG. 12B is output and displayed on the display unit 4.

[0064] The control processing unit 2, input unit 3, display unit 4, IF unit 5, and storage unit 6 can be configured by, for example, a desktop, notebook, tablet, or other computer. If the image acquisition unit 1 is an interface circuit or a communication interface circuit, the IF unit 5 can also serve as the image acquisition unit 1, and therefore the object detection device D, including the image acquisition unit 1, can be configured by a computer.

[0065] Next, the operation of this embodiment will be described. Fig. 13 is a flowchart showing the operation of the object detection device regarding generation of a training dataset. Fig. 14 is a flowchart showing the operation of the object detection device regarding machine learning of an object detection model. Fig. 15 is a flowchart showing the operation of the object detection device regarding detection of a detection target.

[0066] When the object detection device D with the training data generation function configured as described above is powered on, it initializes the necessary parts and starts operation. By executing the control processing program, the control processing unit 2 is functionally configured with a control unit 21, a first machine learning unit 22, a detection unit 23, and a training data generation unit 24, and the training data generation unit 24 is functionally configured with a second machine learning unit 241 and a generation unit 242.

[0067] In generating the training data set (training data set for detection), the object detection device D operates as follows.

[0068] 13, first, the object detection device D with a training data generation function acquires a simulated image training dataset using the control unit 21 of the control processing unit 2 and stores it in the storage unit 6 (S11). When manually generating mask images, the user prepares a simulated image training dataset including a plurality of actual images and a plurality of mask images as shown in FIG. 9 above. The user then inputs this prepared simulated image training dataset to the object detection device D via the input unit 3 or the IF unit 5, and the object detection device D acquires it. Alternatively, when automatically generating mask images, the object detection device D generates a plurality of mask images as described above and acquires the simulated image training dataset.

[0069] Next, the object detection device D uses the second machine learning unit 241 of the learning data generation unit 24 in the control processing unit 2 to machine learn a simulated image generation model based on the simulated image learning dataset (plurality of actual images and plural mask images) acquired in process S11, and stores the machine-learned simulated image generation model in the memory unit 6 (S12).

[0070] Next, the object detection device D acquires, via the control unit 21, multiple pairs of real images and mask images via the input unit 3 or the IF unit 5, and stores them in the storage unit 6 (S13). For example, when the machine learning of the simulated image generation model is completed, the object detection device D causes the control unit 21 to display, on the display unit 4, an image prompting the user to input multiple pairs of real images and mask images. The user refers to this screen and inputs multiple pairs of real images and mask images to the object detection device D via the input unit 3 or the IF unit 5. As described above, the number of real images may be one for each of the multiple mask images, but of course, multiple real images may also be used. In the case of one real image, multiple mask images in the simulated image training dataset acquired in process S11 may be reused, and that one real image may be combined with each of the multiple mask images.

[0071] Next, the object detection device D generates multiple supervised learning data (pairs of mask images and simulated images) by using multiple pairs of actual images and mask images acquired in process S13 in the machine-learned simulated image generation model trained in process S12 by the generation unit 242 of the learning data generation unit 24 in the control processing unit 2, thereby generating a detection learning dataset and storing it in the memory unit 6 (S14).

[0072] Then, the object detection device D causes the control unit 21 to output the detection learning data set generated in process S14 from the IF unit 5 to an external device such as a storage medium or recording medium, and ends this process (S15).

[0073] With respect to generating a training dataset for detection, the object detection device D operates in this manner.

[0074] Regarding the machine learning of the object detection model, the object detection device D operates as follows.

[0075] 14, first, the object detection device D acquires a detection training dataset using the control unit 21 of the control processing unit 2 and stores it in the storage unit 6 (S21). For example, a user inputs the detection training dataset to the object detection device D from the IF unit 5, and the object detection device D acquires it. Alternatively, when this process is executed following each process for generating the detection training dataset shown in FIG. 13 above, the object detection device D acquires the training dataset from the storage unit 6 and stores it in the storage unit 6, so this process S21 can be omitted.

[0076] Next, the object detection device D uses the first machine learning unit 22 of the control processing unit 2 to machine learn an object detection model based on the detection learning data set acquired in step S21 (S22).

[0077] Then, the object detection device D stores the machine-learned object detection model trained in step S22 by the first machine learning unit 22 of the control processing unit 2 in the storage unit 6, and ends this process (S23).

[0078] With respect to the machine learning of the object detection model, the object detection device D operates in this way.

[0079] Regarding detection of a detection target, the object detection device D operates as follows: When detection of a detection target is started, in Fig. 15, the object detection device D causes the control unit 21 of the control processing unit 2 to acquire an image from the image acquisition unit 1 and store the image in the memory unit 6 (S31).

[0080] Next, the object detection device D detects the detection target based on the image acquired from the image acquisition unit 1 in process S31 by using the object detection model machine-learned by the first machine learning unit 22 through the detection unit 23 of the control processing unit 2 (S32).

[0081] Next, the object detection device D causes the detection unit 23 to display and output the detection result of the process S32 on the display unit 4 (S33).

[0082] Then, the object detection device D determines, by the control unit 21, whether or not this process has ended (S34). If the result of this determination is that this process has ended (Yes), the object detection device D ends this process, and if the result of the determination is that this process has not ended (No), the object detection device D returns the process to process S31. Therefore, until this process ends, the object detection device D repeatedly executes each process of detecting the detection target. For example, if, during the execution of each of the above-mentioned processes S31 to S33, the input of a predetermined command instructing the end of the process is received at the input unit 3 (if a predetermined input switch or the like is operated for input), it is determined that this process has ended, and in any other case, it is determined that this process has not ended.

[0083] The object detection device D operates in this manner with respect to the detection of the detection target.

[0084] As described above, the object detection device D with training data generation function, and the training data generation method and training data generation program implemented therein, generate simulated images based on actual images and mask images, and pair the simulated images with the mask images to generate training data using the mask images as training data. The object detection device D with training data generation function, training data generation method, and training data generation program generate a training data set including a plurality of such training data. Therefore, the object detection device D with training data generation function, training data generation method, and training data generation program can further reduce the amount of annotation work.

[0085] The object detection device D with the training data generation function, the training data generation method, and the training data generation program use CycleGAN as a machine learning method, and therefore can appropriately generate a simulated image based on a real image and a mask image so that the background part of the mask image, excluding the target schematic image, becomes the real image.

[0086] The object detection device D with the training data generation function, the training data generation method, and the training data generation program include a detector that distinguishes and detects a background image and an object image from the simulated image, thereby enabling more appropriate generation of the simulated image.

[0087] According to the above, it is possible to provide an object detection device D with a training data generation function that uses a training data set generated by the training data generation unit 24 to machine-train a machine learning model that detects a detection target, as well as an object detection method and object detection program implemented therein. The object detection device D, object detection method, and object detection program include the training data generation unit 24, and therefore can machine-train a first machine learning model (object detection model) without separately preparing a training data set.

[0088] In order to express the present invention, the present invention has been properly and sufficiently described above through the embodiments with reference to the drawings, but it should be recognized that those skilled in the art can easily change and / or improve the above-mentioned embodiments. Therefore, unless the changes or improvements made by those skilled in the art are at a level that causes departure from the scope of the claims described in the claims, such changes or improvements are interpreted as being included in the scope of the claims. [Explanation of symbols]

[0089] D Object detection device with learning data generation function 1 Image acquisition unit 2. Control processing section 3 Input section 4 Display section 5 Interface section (IF section) 6 Memory section 21 Control Unit 22 Machine Learning Department 1 23 Detector 24 Learning data generation unit 241 Second Machine Learning Department 242 Generation part 2411 Generator 2412 Classifier 2413 detector

Claims

1. a predetermined machine learning method is used based on a plurality of real images including a predetermined object and a plurality of mask images each formed by randomly arranging one or more target schematic images that schematically represent the object on a single-color background different from the target schematic images; a plurality of simulated images are generated in which the target schematic images are left as they are in a first image region in the mask image in which the target schematic images are arranged, and a remaining second image region excluding the first image region is replaced with a real image; and a training data set is generated including a plurality of pairs of the mask images of teacher data and the simulated images of training data by combining, for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image; Training data generation device.

2. The machine learning method is CycleGAN. The training data generating device according to claim 1 .

3. The CycleGAN machine learning method includes a detector that distinguishes and detects a background image excluding a first image region of the object from the simulated image and an object image including only the first image region of the object from the simulated image. The training data generating device according to claim 2 .

4. the target object is a detection target, The training data generation device according to any one of claims 1 to 3; a machine learning unit that performs machine learning to create a machine learning model that detects the detection target from an image using a training data set generated by the training data generation device, Object detection device.

5. A computer-executable training data generation method that uses a predetermined machine learning method based on a plurality of real images including a predetermined target object and a plurality of mask images formed by randomly arranging one or more target schematic images that schematically represent the target object on a monochrome background different from the target schematic images, to generate a plurality of simulated images in which a first image region in the mask image in which the target schematic image is arranged is left as is and a remaining second image region excluding the first image region is replaced with a real image, and by combining, for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image into one pair, thereby generating a training data set including a plurality of pairs of the mask image of teacher data and the simulated image of training data.

6. A computer-executable training data generation program that uses a predetermined machine learning method based on a plurality of real images including a predetermined target object and a plurality of mask images formed by randomly arranging one or more target schematic images that schematically represent the target object on a monochrome background different from the target schematic images, to generate a plurality of simulated images in which a first image region in the mask image in which the target schematic image is arranged is left as is and a remaining second image region excluding the first image region is replaced with a real image, and that generates a training data set including a plurality of pairs of the mask image of teacher data and the simulated image of training data by combining, for each of the plurality of mask images, the mask image and the simulated image corresponding to the mask image into one pair.

7. a training data generation process for generating a training data set including a plurality of pairs of the mask image of the training data and the mask image of the training data, by using a predetermined machine learning method based on a plurality of real images including a predetermined object to be detected and a plurality of mask images each including one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images, the plurality of simulated images being generated by leaving the target schematic images as they are in a first image region in which the target schematic images are arranged in the mask images and replacing the remaining second image region excluding the first image region with real images, and forming a pair of the mask image and the simulated image corresponding to the mask image for each of the plurality of mask images; a machine learning step of performing machine learning on a machine learning model that detects the detection target from an image using the training data set generated in the training data generation step, A computer-implemented object detection method.

8. a training data generation process for generating a training data set including a plurality of pairs of the mask image of the training data and the mask image of the training data, by using a predetermined machine learning method based on a plurality of real images including a predetermined object to be detected and a plurality of mask images each including one or more target schematic images that schematically represent the object and are randomly arranged on a single-color background different from the target schematic images, the plurality of simulated images being generated by leaving the target schematic images as they are in a first image region in which the target schematic images are arranged in the mask images and replacing the remaining second image region excluding the first image region with real images, and forming a pair of the mask image and the simulated image corresponding to the mask image for each of the plurality of mask images; a machine learning step of performing machine learning on a machine learning model that detects the detection target from an image using the training data set generated in the training data generation step, An object detection program executed by a computer.

Images