Training data generation device, training data generation method, and training data generation program

Abstract

To provide a learning data generation device which generates a learning data set for machine learning which detects an object by using a remote sensing image.SOLUTION: A learning data generation device 1 comprises: a style conversion learning unit 12 which adjusts a parameter of machine learning which uses a first real image, a first mask image in which an area surrounding an image of an object is masked in the first real image, and a first simulated image showing the image of the object to generate a combined image showing the first simulated image and a background matching an unmasked area, as an image to be embedded to the masked area; and a second learning data generation unit 13 which generates a second mask image by masking a range to which a second simulated image showing the image of the object is to be embedded, in a second real image and generates a combined image showing the second simulated image and the background as an image to be embedded to the masked area on the basis of machine learning using the adjusted parameter, the second simulated image, and the second mask image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a training data generation device, a training data generation method, and a training data generation program. [Background technology]

[0002] Object detection is a process of detecting a specific object in an image, and is realized by means of machine learning, etc. Specific examples of the object include a human being, a building, or a vehicle.

[0003] Object detection ML (Machine Learning) uses a set of an image and data indicating the position of an object in the image as input, performs a process called learning, and outputs a trained model that can detect objects. Here, the data is called annotation. The set is called a learning dataset.

[0004] In object detection using synthetic aperture radar images, the scattering cross section may vary significantly depending on the angle (azimuth angle and incident angle) of the synthetic aperture radar device relative to the target. Therefore, in order to generate a trained model with relatively high accuracy, it is necessary to prepare a wide variety of angle patterns in the training dataset. However, due to the effort and frequency of imaging, the training dataset may lack angle patterns based on actual synthetic aperture radar images.

[0005] To address the lack of angle patterns based on actual synthetic aperture radar images, Patent Document 1 discloses a method of generating simulated images, converting the generated simulated images into images that match the actual images, and using the style-converted simulated images as a training data set in addition to the actual synthetic aperture radar images. The simulated images are generated by simulating the reflection of radio waves from an object based on the model information of the object. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2021 / 038678 Brochure Summary of the Invention [Problem to be solved by the invention]

[0007] In the method disclosed in Patent Document 1, when a simulated image is used as a learning dataset, two types of background addition processing and one type of embedding processing are performed. The background addition processing is a processing of adding a background derived from a real image to the simulated image. The embedding processing is a processing of embedding the simulated image with the background added into the real image. However, in the method disclosed in Patent Document 1, there is a problem that a boundary line that cannot exist in the real image is mixed into the simulated image in the background addition processing and the embedding processing. As a result, the simulated image becomes an image that is not similar in appearance to the real image, and the accuracy of style conversion and final object detection is deteriorated.

[0008] The first type of background addition process is a process for generating data to be learned by pix2pix, which is a type of generative adversarial network. In the method disclosed in Patent Document 1, a prediction model for converting a simulated image into a style that harmonizes with the real image is obtained by having pix2pix learn an image pair (a pair of a simulated image and a real image). However, as a pre-processing for obtaining a prediction model, a real image is superimposed around an object in the simulated image in order to increase the similarity of the image pair and improve the accuracy of style conversion. Therefore, the first type of background addition process has a problem that a rectangular boundary that cannot exist in the real image is mixed into the simulated image by superimposing the real image. As a result, the learning of pix2pix is ​​hindered.

[0009] The second type of background addition process is a process of adding a background to a simulated image before performing style conversion. In the background addition process, a background image extracted from a real image is superimposed on the outer part of a rectangular area of ​​an object in the simulated image. Therefore, the second type of background addition process has a problem that a rectangular boundary line that does not exist in the real image is mixed into the simulated image by superimposing the background image. As a result, the accuracy of style conversion by a predictive model is deteriorated.

[0010] The embedding process is a process of embedding a style-converted simulated image into a real image. In the embedding process, an image that serves as a learning dataset for object detection ML is created by superimposing the style-converted simulated image on the real image. Therefore, the embedding process has a problem that a rectangular boundary line that cannot exist in the real image is mixed in the boundary between the style-converted simulated image and the real image in the simulated image by superimposing the simulated image on the real image. As a result, the learning of object detection ML is hindered.

[0011] The present disclosure aims to prevent boundaries that cannot exist in an actual image from being mixed into a simulated image when generating a learning dataset for machine learning that performs object detection using remote sensing images. [Means for solving the problem]

[0012] The training data generating device according to the present disclosure comprises: A learning data generation device that generates a learning dataset for machine learning to detect an object from a remote sensing image, comprising: a style conversion learning unit that performs a first machine learning process using at least one first real image, which is an image containing an image of the object and which is actually captured by remote sensing, at least one first mask image generated by masking an area surrounding the image of the object in each of the at least one first real image, and at least one first simulated image generated by superimposing an image of the object corresponding to each of the at least one first real image on a base image, to generate a first synthetic image showing each of the first simulated image corresponding to each of the at least one first mask image and a background that harmonizes with the unmasked area as an image to be embedded in the masked area in each of the at least one first mask image, and adjusts parameters in the first machine learning process; a second learning data generation unit that generates a second mask image by masking a region indicated by embedding range information indicating a range in which a second simulated image indicating an image of the object is to be embedded in a second actual image which is an image actually captured by remote sensing, and generates a second synthetic image indicating each of the second simulated image and a background based on a second machine learning that is equivalent to the first machine learning and that uses the adjusted parameters, the second simulated image, and the second mask image as an image to be embedded in the masked region of the generated second mask image; Equipped with. Effect of the Invention

[0013] According to the present disclosure, the style conversion learning unit adjusts parameters in the first machine learning by performing a first machine learning, which is machine learning that generates a first synthetic image consisting of a first simulated image corresponding to an image actually captured by remote sensing and a background that harmonizes with an unmasked area of ​​the first mask image corresponding to an image actually captured by remote sensing. In addition, the second learning data generating unit performs a second machine learning using the second simulated image, the second mask image, and the adjusted parameters to generate a second synthetic image having a background that harmonizes with an unmasked area of ​​the second mask image. Here, the second machine learning is equivalent to the first machine learning. That is, when the second synthetic image is embedded in the second mask image, no boundary line that cannot exist in the real image is generated. Therefore, according to the present disclosure, when generating a learning dataset for machine learning that performs object detection using a remote sensing image, it is possible to prevent a boundary line that cannot exist in the real image from being mixed into the simulated image. [Brief description of the drawings]

[0014] [Figure 1] 1 is a diagram showing an example of the configuration of a training data generating device 1 according to a first embodiment. [Diagram 2] 2 is a conceptual diagram illustrating a wide-area actual image 21, annotation data 22, and imaging condition information 23 according to the first embodiment. [Diagram 3] 3 is a conceptual diagram for explaining a background-added simulated image group 3 according to the first embodiment. FIG. [Figure 4] 2 is a conceptual diagram illustrating each of object range information 41, an actual image 42, a mask image 43, and a simulation image 44 according to the first embodiment. [Diagram 5] FIG. 2 is a diagram for explaining a generative adversarial network according to the first embodiment. [Figure 6] 2 is a conceptual diagram illustrating embedding range information 51, an actual image 52, a mask image 53, and a simulation image 54 according to the first embodiment. [Figure 7] 1 is a diagram showing an example of a hardware configuration of a training data generating device 1 according to a first embodiment. [Figure 8]3 is a flowchart showing the operation of the training data generation device 1 according to the first embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a hardware configuration of a training data generation device 1 according to a modification of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In the description of the embodiments and the drawings, the same elements and corresponding elements are given the same reference numerals. Descriptions of elements given the same reference numerals are omitted or simplified as appropriate. Arrows in the drawings primarily indicate data flow or processing flow. In addition, "part" may be read as "circuit," "step," "procedure," "processing," or "circuitry" as appropriate.

[0016] Embodiment 1 Hereinafter, the present embodiment will be described in detail with reference to the drawings.

[0017] ***Configuration Description*** First, an example of the overall configuration of a training data generating device 1 will be described with reference to FIGS. FIG. 1 is a block diagram showing an example of the configuration of a training data generation device 1. The training data generation device 1 generates a training data set for machine learning to detect an object from a remote sensing image. As shown in FIG. 1, the training data generation device 1 receives a wide-area actual image 21, annotation data 22, and imaging condition information 23 as input, and outputs a background-added simulated image group 3. The training data generation device 1 includes a first training data generation unit 11, a style conversion learning unit 12, and a second training data generation unit 13. The first training data generation unit 11 is also called a style conversion training data generation unit. The second training data generation unit 13 is also called an object detection training data generation unit.

[0018] FIG. 2 is a conceptual diagram for explaining specific examples of a wide-area actual image 21, annotation data 22, and imaging condition information 23. As shown in FIG. The wide-area actual image 21 is an image obtained as a result of imaging by a synthetic aperture radar (hereinafter referred to as SAR), and is a grayscale image having, for each pixel, information indicating the amplitude of a received signal of a radio wave reflected by a feature, and includes at least one of an image 211 of an object and an image 212 of a background. The image 211 of an object is an image showing the reflection of the object in the wide-area actual image 21. The image 212 of the background is an image showing the reflection of the background in the wide-area actual image 21. The background is a feature other than the object. The annotation data 22 is graphic data in which the range in which each object exists in the wide-area actual image 21 is expressed by a rectangle 221. As the annotation data 22, teacher data prepared for learning of object detection ML (Machine Learning) may be diverted. The imaging condition information 23 is information indicating the imaging conditions when imaging was performed by the SAR, and specifically includes information indicating each of an incident angle 231, a transmission wave polarization direction 232, and a reception wave polarization direction 233.

[0019] As a specific example, a SAR device mounted on a flying object irradiates radio waves toward the ground, receives the radio waves reflected by the background or an object, and generates a wide-area actual image 21 based on the reception signal indicated by the received radio waves. Specific examples of the flying object include an artificial satellite, an airplane, or a drone. A target object is an object that exists at a lower altitude than the flying object, and specific examples of the target object include an airplane, a ship, a train, or a vehicle.

[0020] 3 is a conceptual diagram for explaining a specific example of the background-added simulated image group 3. The background-added simulated image group 3 is made up of at least one background-added simulated image 31. As a specific example, the background-added simulated image 31 is a standard image in which an image of a simulated object is generated by simulating the incidence and reflection of radio waves in consideration of the shape, material, and azimuth angle of the object, and the imaging condition information 23, and the generated image is added with the background of a real image. The background-added simulated image 31 is an image that has been style-converted so that the entire image blends with the actual SAR image. Here, the standard image is an image that can be input to the object detection ML at once because the width and height of the image satisfy predetermined conditions. Blending with the SAR image means that the difference in appearance from the actual SAR image is relatively small. The style conversion of the entire image of the background-added simulated image 31 to blend with the actual SAR image may mean style conversion of only the second synthetic image, or only the background of the second synthetic image to blend with the unmasked area.

[0021] In SAR, when the angle of the object from the SAR device is different, the appearance of the object may change significantly between images in which the object is captured. The angle indicates the orientation of the object, and is at least one of the azimuth angle and the angle of incidence. Therefore, in order to detect the object from the actual SAR image with relatively high accuracy using machine learning, SAR images of the object captured at various angles are required as a learning data set. However, due to the effort and frequency of capturing images, it is difficult to prepare SAR images captured at various angles. Therefore, the learning data set derived only from the wide-area actual image 21 cannot ensure the accuracy of detecting the object.

[0022] Meanwhile, the learning data generation device 1 can generate a large number of background-added simulated images 31 using a data set consisting of one each of the wide-area actual image 21, the annotation data 22, and the imaging condition information 23 as input. Here, the learning data generation device 1 can include, in the large number of background-added simulated images 31 to be generated, an image of an object at an angle different from the angle in the wide-area actual image 21. By using the background-added simulated image 31 including the image of an object at an angle different from the angle in the wide-area actual image 21 as a learning data set for object detection ML, it is expected that the accuracy of object detection in object detection based on object detection ML will be improved.

[0023] Next, a detailed configuration example of the training data generation device 1 will be described with reference to FIGS. 1, 4, 5, and 6. FIG.

[0024] As shown in FIG. 1, the first learning data generation unit 11 includes an annotation data reading unit 111, an image clipping unit 112, a mask processing unit 113, and a simulation image generation unit 114. The annotation data reading unit 111 reads the annotation data 22, generates object range information 41 based on the annotation data 22 that has been read, and outputs the generated object range information 41. The image cropping unit 112 receives each of the wide area real image 21 and the object range information 41, generates a real image 42 by cropping an image including an object image 421 from the received wide area real image 21 based on the received object range information 41, and outputs the generated real image 42. The real image 42 includes the object image 421. The mask processing unit 113 receives each of the object range information 41 and the actual image 42, and generates a mask image 43 by superimposing random noise on a rectangular range including the image 421 of the object based on the received object range information 41 on the received actual image 42, and outputs the generated mask image 43. The rectangle may be inclined. The simulated image generation unit 114 receives each of the imaging condition information 23 and the object range information 41, and generates a simulated image 44 under the same imaging conditions and azimuth angle as the imaging conditions and azimuth angle corresponding to the image 421 of the object contained in the actual image 42 based on the received imaging condition information 23 and object range information 41, and outputs the generated simulated image 44.

[0025] FIG. 4 is a conceptual diagram for explaining specific examples of object range information 41, actual image 42, mask image 43, and simulation image 44.

[0026] The object range information 41 is data generated from the annotation data 22. The object range information 41 includes information indicating each of an azimuth angle 412, a cutout X coordinate 413, a cutout Y coordinate 414, an object X coordinate 415, an object Y coordinate 416, a vertical width 417, and a horizontal width 418. The object range 411 is a rectangular region surrounding the object image 421. The object range 411 is defined by an azimuth angle 412, a vertical width 417, and a horizontal width 418. Azimuth angle 412 is the inclination of the object in the clockwise direction when the upward direction of the image is set to 0 degrees, and represents the inclination of the direction of the front of the object relative to the rear of the object. The cut-out X coordinate 413 and the cut-out Y coordinate 414 represent the position of the upper left corner of the cut-out range 419 . Object X coordinate 415 and object Y coordinate 416 represent the relative center position of object image 211 . Vertical width 417 indicates the width of the object in the front-to-rear direction. Width 418 indicates the width of the object in the left-right direction. The cut-out range 419 indicates the width and height of the cut-out range. Note that the width and height of the cut-out range 419 are each a rated size.

[0027] The real image 42 is a standard image cut out from the wide-area real image 21 based on the cut-out range 419, and includes an image 421 of the object and an image 422 of the background.

[0028] The mask image 43 is an image in which random noise 431 is superimposed on the object range 411 in the actual image 42, and is a nominal image.

[0029] The simulated image 44 is an image on which an image of a simulated object is superimposed, and is a standard image. The image of the simulated object is an image generated by simulating the incidence and reflection of radio waves on the object based on the imaging condition information 23 and the azimuth angle 412 indicated by the object range information 41.

[0030] As shown in FIG. 1, the style conversion learning unit 12 receives each of the real image 42, the mask image 43, and the simulated image 44, generates the complementary network parameters 124 based on the received real image 42, the mask image 43, and the simulated image 44, and outputs the generated complementary network parameters 124. The real image 42 is also called the first real image. The mask image 43 is also called the first mask image. The simulated image 44 is also called the first simulated image. The style conversion learning unit 12 adjusts the parameters in the machine learning by performing machine learning to generate a first synthetic image as an image to be embedded in a masked region in each of the at least one mask image 43 using at least one real image 42, at least one mask image 43, and at least one simulated image 44. Each of the at least one real image 42 is an image in which an image 421 of an object is captured, and is an image actually captured by remote sensing. Each of the at least one mask image 43 is an image generated by masking a region surrounding the image 421 of the object in each of the at least one real image 42. In each of the at least one mask image 43, the masked region may be a region in which random noise 431 is superimposed. Each of the at least one simulated image 44 is an image generated by superimposing the image 421 of the object corresponding to each of the at least one real image 42 on a base image. The base image is typically a plain image. The machine learning performed by the style conversion learning unit 12 is also called first machine learning. The shape of the region surrounding the image 421 of the object is typically rectangular. The first composite image is an image showing each of the simulated image 44 corresponding to each of the at least one mask image 43 and a background that blends with the unmasked region of the mask image 43. The image showing the simulated image 44 may be an image in which the simulated image 44 is modified. The size of the first composite image is the size of the masked region.When each of at least one simulated image 44 is a target first simulated image and a real image 42 corresponding to the target first simulated image is a target first real image, the target first simulated image is an image generated by simulating the incidence and reflection of electromagnetic waves in the target based on the imaging conditions corresponding to the target first real image and the orientation of the image 421 of the target in the target first real image. A specific example of the electromagnetic wave is radio waves. The orientation of the image 421 of the target in the target first real image and the area surrounding the image 421 of the target in the target first real image may be determined based on the annotation data 22. The annotation data 22 indicates the orientation of the image 421 of the target in the target first real image and the area in which the target exists. The style transfer learning unit 12 acquires the ability to synthesize two images so as to match the actual SAR image by performing supervised learning. In the supervised learning, the input of the supervised data is the received mask image 43 and the simulated image 44, and the output is the received actual image 42. The supervised learning performed by the style transfer learning unit 12 may be learning in a generative adversarial network format using three neural networks, namely, the complementary network 61, the global discriminant network 62, and the local discriminant network 63 shown in FIG. 5. Hereinafter, a case where the style transfer learning unit 12 performs learning in a generative adversarial network format will be described. Also, an image in which a first synthetic image corresponding to at least one mask image 43 is embedded in at least one mask image 43 is set as a target embedded image.

[0031] The interpolation network 61 is a neural network that generates one background-added simulated image 66 by synthesizing two images, the simulated image 44 and the mask image 43. That is, the interpolation network 61 generates a first synthesized image corresponding to each of at least one mask image 43. Both the global classification network 62 and the local classification network 63 are neural networks that receive either the background-added simulated image 66 or the real image 42 as input, and output classification information 65. The classification information 65 is information that indicates whether the input was the background-added simulated image 66 or the real image 42. Here, the global classification network 62 receives the entire background-added simulated image 66 or the entire real image 42 as input. The local classification network 63 receives the patch 64 as input. The patch 64 is a cutout of the background-added simulated image 66 and the real image 42, and is an area of ​​the background-added simulated image 66 and the real image 42 that is represented by a rectangle that includes the image of the object. The global classification network 62 receives at least one of the real images 42 or a target embedded image as input, and outputs a result of classifying whether the input is a real image 42 or not. The local classification network 63 receives as input an image of a region surrounding the image 421 of the object in at least one real image 42, or an image of a region surrounding the image of the object in the object-embedded image, and outputs a result of classifying whether the input is part of the real image 42 or not. The style transfer learning unit 12 adjusts the parameters of the complementation network 61 so that the global classification network 62 and the local classification network 63 tend to output erroneous results. In addition, the style transfer learning unit 12 adjusts the parameters of each of the global classification network 62 and the local classification network 63 so that the accuracy of each of the global classification network 62 and the local classification network 63 is increased.

[0032] The parameters of these three neural networks in the style transfer learning unit 12 are adjusted by a generative adversarial network. Specifically, the parameters of the complementation network 61 are adjusted so as to output an image that is difficult to distinguish by the global classification network 62 and the local classification network 63. In addition, the weights of the global classification network 62 and the local classification network 63 are adjusted so that the global classification network 62 and the local classification network 63 can more accurately distinguish whether an input image is the background-added simulated image 66 or the real image 42, respectively.

[0033] As a result of adjusting the parameters of the three neural networks, the background-added simulated image 66 output from the complementation network 61 becomes an image that the global classification network 62 and the local classification network 63 mistake for a real image. Specifically, in the background-added simulated image 66, an image of an object derived from a simulation calculation is embedded in the portion of the mask image 43 where the random noise 431 is superimposed, and the entire image is converted into a style that matches the actual SAR image. In addition, the rectangular boundary line that appears in the method of Patent Document 1 does not appear in the background-added simulated image 66.

[0034] As shown in FIG. 1, the second learning data generating unit 13 includes an embedding range determining unit 131, an image cropping unit 132, a simulated image generating unit 134, a mask processing unit 133, and a complementation network 135. The second learning data generating unit 13 generates a mask image 53 by masking a region indicated by embedding range information 51 in a real image 52, and generates a second synthetic image based on machine learning using the adjusted parameters, the simulated image 54, and the mask image 53 as an image to be embedded in the masked region of the generated mask image 53. The real image 52 is an image actually captured by remote sensing, and is also called a second real image. The mask image 53 is also called a second mask image. The simulated image 54 indicates an image of an object, and is also called a second simulated image. The embedding range information 51 indicates a range in which the simulated image 54 is embedded. In the second mask image, the masked region may be a region in which random noise 531 is superimposed. The machine learning performed by the second learning data generating unit 13 is also called second machine learning. The second machine learning is equivalent to the first machine learning. Each of the algorithms of the first machine learning and the second machine learning may be a generative adversarial network. The second synthetic image is an image showing each of the simulated image 54 and the background. The image showing the simulated image 54 may be an image obtained by correcting the simulated image 54. The background shown by the second synthetic image blends with the unmasked area of ​​the mask image 53. The size of the second synthetic image is the same as the size of the masked area. The second learning data generating unit 13 may use the result of adjusting the parameters of the complementation network 61 in the second machine learning. The second learning data generating unit 13 generates the simulated image 54 by simulating the incidence and reflection of electromagnetic waves in the target object based on the imaging conditions corresponding to the real image 52 and the orientation indicated by the embedding range information 51. The embedding range determination unit 131 generates embedding range information 51 based on random numbers, and outputs the generated embedding range information 51. The embedding range information 51 is numerical data generated using random numbers, and includes information indicating the orientation of the object. The image cropping unit 132 receives each of the wide-area real image 21 and the embedding range information 51, generates a real image 52 by cropping the received wide-area real image 21 based on the received embedding range information 51, and outputs the generated real image 52. The mask processing unit 133 receives each of the embedding range information 51 and the actual image 52, generates a mask image 53 by masking a portion of the received actual image 52 with random noise 531 in accordance with the received embedding range information 51, and outputs the generated mask image 53. The simulated image generating unit 134 receives the imaging condition information 23 and the embedding range information 51, generates a simulated image 54 based on the received imaging condition information 23 and embedding range information 51, and outputs the generated simulated image 54. The simulated image 54 is an image of an object captured under the same imaging conditions as those corresponding to the wide-area actual image 21, and corresponds to an image of the object captured from a random angle. The interpolation network 135 uses the interpolation network parameters 124 output by the style conversion learning unit 12, receives the mask image 53 and the simulated image 54 as input, and outputs the background-added simulated image 31. In terms of input and output formats and processing contents, the image cropping unit 132, the mask processing unit 133, and the simulation image generation unit 134 are equivalent to the components of the same name provided in the first training data generation unit 11. In terms of input and output formats and processing contents, the complementation network 135 is equivalent to the complementation network 61.

[0035] FIG. 6 is a conceptual diagram for explaining each of embedding range information 51, actual image 52, mask image 53, and simulation image .

[0036] The embedding range information 51 includes information indicating each of a cutout X coordinate 511 , a cutout Y coordinate 512 , an object X coordinate 513 , an object Y coordinate 514 , an azimuth angle 515 , a vertical width 516 , and a horizontal width 517 . Random values ​​are selected for each of cutout X coordinate 511, cutout Y coordinate 512, object X coordinate 513, object Y coordinate 514, and azimuth angle 515. The vertical width 516 and the horizontal width 517 are respectively the vertical width 417 and the horizontal width 418 corresponding to the object image 421. In other words, each of the vertical width 516 and the horizontal width 517 is a known value. The width and height of the cutout range 518 are nominal.

[0037] The actual image 52 is an image cut out from the wide-area actual image 21 based on the embedding range information 51 .

[0038] The mask image 53 is an image in which random noise 531 is superimposed on the actual image 52 based on the embedding range information 51, and is a standard image.

[0039] The simulated image 54 is an image on which an image of a simulated object is superimposed, and is a standard image. The image of the simulated object is an image generated by simulating the incidence and reflection of radio waves on the object based on the imaging condition information 23 and the embedding range information 51.

[0040] 7 shows an example of a hardware configuration of the training data generation device 1 according to this embodiment. The training data generation device 1 is made up of a computer. The training data generation device 1 may be made up of multiple computers.

[0041] As shown in the figure, the training data generation device 1 is a computer including hardware such as a processor 91, a memory 92, an auxiliary storage device 93, an input / output IF (Interface) 94, and a communication device 95. These pieces of hardware are appropriately connected via signal lines 99.

[0042] The processor 91 is an integrated circuit (IC) that performs arithmetic processing and controls the hardware of the computer. Specific examples of the processor 91 include a central processing unit (CPU), a digital signal processor (DSP), or a graphics processing unit (GPU). The training data generation device 1 may include a plurality of processors that replace the processor 91. The plurality of processors share the role of the processor 91.

[0043] The memory 92 is typically a volatile storage device, and a specific example is a RAM (Random Access Memory). The memory 92 is also called a primary storage device or a main memory. Data stored in the memory 92 is saved in the auxiliary storage device 93 as necessary.

[0044] The auxiliary storage device 93 is typically a non-volatile storage device, and specific examples thereof include a ROM (Read Only Memory), an HDD (Hard Disk Drive), or a flash memory. Data stored in the auxiliary storage device 93 is loaded into the memory 92 as necessary. The memory 92 and the auxiliary storage device 93 may be integrated into one unit.

[0045] The input / output IF 94 is a port to which an input device and an output device are connected. A specific example of the input / output IF 94 is a Universal Serial Bus (USB) terminal. Specific examples of the input device are a keyboard and a mouse. A specific example of the output device is a display.

[0046] The communication device 95 is a receiver and a transmitter, and is, for example, a communication chip or a network interface card (NIC).

[0047] Each unit of the training data generation device 1 may use the input / output IF 94 and the communication device 95 as appropriate when communicating with other devices.

[0048] The auxiliary storage device 93 stores a training data generation program. The training data generation program is a program that causes a computer to realize the functions of each unit of the training data generation device 1. The training data generation program is loaded into the memory 92 and executed by the processor 91. The functions of each unit of the training data generation device 1 are realized by software.

[0049] Data used when executing the learning data generation program and data obtained by executing the learning data generation program are appropriately stored in a storage device. Each part of the learning data generation device 1 appropriately uses a storage device. As a specific example, the storage device is composed of at least one of the memory 92, the auxiliary storage device 93, a register in the processor 91, and a cache memory in the processor 91. Note that data and information may have the same meaning. The storage device may be independent of the computer. The functions of the memory 92 and the auxiliary storage device 93 may be realized by other storage devices.

[0050] The training data generation program may be recorded in a computer-readable non-volatile recording medium. Specific examples of the non-volatile recording medium include an optical disk and a flash memory. The training data generation program may be provided as a program product.

[0051] ***Explanation of Operation*** The operation procedure of the training data generating device 1 corresponds to a training data generating method, and the program that realizes the operation of the training data generating device 1 corresponds to a training data generating program.

[0052] Next, a learning data generating method will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the learning data generating method, and shows a series of processes until the learning data generating device 1 generates the background-added simulated image group 3.

[0053] When the wide-area actual image 21, the annotation data 22, and the imaging condition information 23 are input to the first learning data generation unit 11, the first learning data generation unit 11 starts executing the process.

[0054] (Step S701) Annotation data reading unit 111 reads annotation data 22, generates object range information 41 based on the read annotation data 22, and outputs the generated object range information 41. Here, values ​​are randomly selected for each of cut-out X coordinate 413 and cut-out Y coordinate 414 such that object range 411 is included in cut-out range 419. Note that object range 411 includes one object image 421.

[0055] (Step S702) The image cropping unit 112 generates an actual image 42 by cropping out a nominal range from the actual wide-area image 21, with the position corresponding to the cropping X coordinate 413 and the cropping Y coordinate 414 indicated by the object range information 41 as its upper left corner.

[0056] (Step S703) The mask processing unit 113 generates a mask image 43 by superimposing random noise 431 on a rectangular area in the actual image 42 that corresponds to the azimuth angle 412, object X coordinate 415, object Y coordinate 416, vertical width 417, and horizontal width 418 indicated by the object range information 41, and that includes an image 421 of the object in the actual image 42.

[0057] (Step S704) The simulated image generation unit 114 generates a simulated image of the object by simulating the incidence and reflection of radio waves at the object based on the imaging condition information 23 corresponding to the actual image 42 and the azimuth angle 412 indicated by the object range information 41, and generates a simulated image 44 by superimposing the generated image on the base image at a position corresponding to the object X coordinate 415 and the object Y coordinate 416.

[0058] (Step S705) The first training data generation unit 11 checks whether a certain number of sets each consisting of an actual image 42, a mask image 43, and a simulation image 44 have been generated. If the set has not been generated a certain number of times, the training data generation device 1 returns to step S701. If the set has been generated a certain number of times, the training data generation device 1 proceeds to step S706. Here, the certain number is, for example, a parameter set by the user for the first training data generation unit 11.

[0059] (Step S706) The style transfer training unit 12 performs training based on a generative adversarial network using a set of a certain number of real images 42, mask images 43, and simulated images 44. The training is mini-batch training. In the training, the training unit 12 adjusts the training network parameters 124, which are parameters of the training network 61, so that an image that is difficult to distinguish by the global training network 62 and the local training network 63 is output from the training network 61. The style transfer training unit 12 also adjusts the parameters of the global training network 62 and the local training network 63 so that the global training network 62 and the local training network 63 can more accurately distinguish whether an image input to each of the global training network 62 and the local training network 63 is the background-added simulated image 66 or the real image 42. The style conversion training unit 12 performs training using a certain number of sets a certain number of times, and then outputs the adjusted complementation network parameters 124.

[0060] When the interpolation network parameters 124 are output from the style conversion learning unit 12, the second learning data generating unit 13 starts executing the process. Note that the wide-area actual image 21 and imaging condition information 23 are input to the second learning data generating unit 13.

[0061] (Step S707) Embedding range determination unit 131 generates embedding range information 51 by setting random values ​​to cutout X coordinate 511, cutout Y coordinate 512, object X coordinate 513, object Y coordinate 514, and azimuth angle 515, respectively.

[0062] (Step S708) The image cropping unit 132 generates an actual image 52 by cropping a nominal range from the actual wide-area image 21, with the position corresponding to the cropping X coordinate 511 and the cropping Y coordinate 512 indicated by the embedding range information 51 as its upper left corner.

[0063] (Step S709) The mask processing unit 133 generates a mask image 53 by superimposing random noise 531 onto a rectangular area in the actual image 52 corresponding to the object X coordinate 513, object Y coordinate 514, azimuth angle 515, vertical width 516, and horizontal width 517 indicated by the embedding range information 51.

[0064] (Step S710) The simulated image generation unit 134 generates a simulated image of the object by simulating the incidence and reflection of radio waves at the object based on the imaging condition information 23 corresponding to the actual image 52 and the azimuth angle 515 indicated by the embedding range information 51, and generates a simulated image 54 by superimposing the generated image on the base image at a position corresponding to the object X coordinate 513 and the object Y coordinate 514.

[0065] (Step S711) The interpolation network 135 generates the background-added simulated image 31 by combining the mask image 53 and the simulated image 54 using the interpolation network parameters 124 .

[0066] (Step S712) The second learning data generating unit 13 checks whether a certain number of background-added simulated images 31 have been generated. If the certain number of background-added simulated images 31 have not been generated, the second learning data generation unit 13 returns to step S707. If the certain number of background-added simulated images 31 have been generated, the learning data generation device 1 ends the process.

[0067] ***Explanation of the Effects of the First Embodiment*** According to this embodiment, the complement network parameters 124 are parameters adjusted by the style conversion learning unit 12. Therefore, in the background-added simulated image 31, which is the output of the complement network 135, an image of an object originating from the simulated image is embedded in a portion where the random noise 531 is superimposed in the mask image 53, and the entire image is converted into a style that blends in with the actual SAR image. Here, when an image with low similarity to the actual SAR image is used as a learning dataset for object detection ML, the accuracy of object detection deteriorates. However, the background-added simulated image 31 is converted into a style that blends in with the actual SAR image as a whole. Therefore, by using the background-added simulated image 31 according to this embodiment as a learning dataset in machine learning that performs object detection using remote sensing images, it is expected that the accuracy of object detection based on the machine learning will be improved. Patent Document 1 also proposes a method in which a small image, which is a real image in which a portion of an image containing an object is masked, is used as a background image, and the background image is associated with a simulated image. However, this method has a problem in that the work of creating a mask corresponding to the small image is cumbersome. On the other hand, according to the present embodiment, it is not necessary to create a mask corresponding to the small image.

[0068] ***Other configurations*** <Variation 1> FIG. 9 shows an example of the hardware configuration of a training data generating device 1 according to this modified example. The training data generation device 1 includes a processing circuit 98 in place of the processor 91 , the processor 91 and a memory 92 , the processor 91 and an auxiliary storage device 93 , or the processor 91 , the memory 92 , and the auxiliary storage device 93 . The processing circuitry 98 is hardware that realizes at least a part of each unit included in the training data generation device 1. The processing circuitry 98 may be dedicated hardware, or may be a processor that executes programs stored in the memory 92 .

[0069] When processing circuitry 98 is dedicated hardware, processing circuitry 98 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The training data generation device 1 may include a plurality of processing circuits that replace the processing circuit 98. The plurality of processing circuits share the role of the processing circuit 98.

[0070] In the training data generation device 1, some of the functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0071] Processing circuitry 98 is illustratively implemented in hardware, software, firmware, or a combination thereof. The processor 91, memory 92, auxiliary storage device 93, and processing circuit 98 are collectively referred to as the “processing circuitry.” In other words, the functions of the functional components of the training data generation device 1 are realized by the processing circuitry.

[0072] ***Other embodiments*** Although the first embodiment has been described, it is possible to combine a plurality of parts of the present embodiment. Alternatively, it is possible to partially implement the present embodiment. In addition, the present embodiment may be modified in various ways as necessary, and may be implemented in any combination, either as a whole or in part. The above-described embodiment is essentially a preferred example, and is not intended to limit the scope of the present disclosure, its applications, and uses. The procedures described using the flowcharts and the like may be modified as appropriate.

[0073] Various aspects of the present disclosure are summarized below as appendices. (Appendix 1) A learning data generation device that generates a learning dataset for machine learning to detect an object from a remote sensing image, comprising: a style conversion learning unit that performs a first machine learning process using at least one first real image, which is an image containing an image of the object and which is actually captured by remote sensing, at least one first mask image generated by masking an area surrounding the image of the object in each of the at least one first real image, and at least one first simulated image generated by superimposing an image of the object corresponding to each of the at least one first real image on a base image, to generate a first synthetic image showing each of the first simulated image corresponding to each of the at least one first mask image and a background that harmonizes with the unmasked area as an image to be embedded in the masked area in each of the at least one first mask image, and adjusts parameters in the first machine learning process; a second learning data generation unit that generates a second mask image by masking a region indicated by embedding range information indicating a range in which a second simulated image indicating an image of the object is to be embedded in a second actual image which is an image actually captured by remote sensing, and generates a second synthetic image indicating each of the second simulated image and a background based on a second machine learning that is equivalent to the first machine learning and that uses the adjusted parameters, the second simulated image, and the second mask image as an image to be embedded in the masked region of the generated second mask image; A learning data generating device comprising: (Appendix 2) each of the first machine learning algorithm and the second machine learning algorithm is a generative adversarial network; In the first machine learning, the style conversion learning unit a global classification network which is a neural network that receives an input of one of the at least one real images or the target-embedded image and outputs a result of classifying whether or not the input is a real image; and a local classification network which is a neural network that receives an input of an image of a region surrounding an image of the object in one of the at least one real images or an image of a region surrounding an image of the object in the target-embedded image and outputs a result of classifying whether or not the input is a part of a real image, Adjusting parameters of the complementary network so that the global discriminant network and the local discriminant network tend to output erroneous results; Adjusting parameters of the global identification network and the local identification network so that the accuracy of each of the global identification network and the local identification network is high; The training data generation device according to claim 1, wherein the second training data generation unit uses a result of adjusting parameters of the complementation network in the second machine learning. (Appendix 3) when each of the at least one first simulated image is defined as a first simulated image of a target, and a first real image corresponding to the first simulated image of the target is defined as a first real image of a target, the first simulated image of the target is an image generated by simulating the incidence and reflection of an electromagnetic wave on the target based on an imaging condition corresponding to the first real image of the target and a direction of an image of the target in the first real image of the target, the embedding range information includes information indicating a direction of the object; The learning data generation device according to claim 1 or 2, wherein the second learning data generation unit generates the second simulated image by simulating incidence and reflection of electromagnetic waves at the object based on imaging conditions corresponding to the second actual image and an orientation of the object indicated by the embedding range information. (Appendix 4) The learning data generation device described in Appendix 3, wherein the orientation of the image of the object in the first real image of the object and the area surrounding the image of the object in the first real image of the object are each determined based on annotation data indicating the orientation of the image of the object in the first real image of the object and the area in which the object exists, respectively. (Appendix 5) In each of the at least one first mask image, the masked region is a region onto which random noise is superimposed; 5. The training data generation device according to claim 1, wherein in the second mask image, a masked region is a region onto which random noise is superimposed. [Explanation of symbols]

[0074] 1 Learning data generation device, 11 First learning data generation unit, 111 Annotation data reading unit, 112 Image clipping unit, 113 Mask processing unit, 114 Simulation image generation unit, 12 Style conversion learning unit, 124 Complement network parameters, 13 Second learning data generation unit, 131 Embedding range determination unit, 132 Image clipping unit, 133 Mask processing unit, 134 Simulation image generation unit, 135 Complement network, 21 Wide-area real image, 211 Image of object, 212 Image of background, 22 Annotation data, 221 Rectangle, 23 Imaging condition information, 231 Incident angle, 232 Transmission wave polarization direction, 233 Reception wave polarization direction, 3 Background-added simulated image group, 31 Background-added simulated image, 41 Object range information, 411 Object range, 412 Azimuth angle, 413 Cutout X coordinate, 414 Cutout Y coordinate, 415 Object X coordinate, 416 Object Y coordinate, 417 Vertical width, 418 Horizontal width, 419 Cutout range, 42 Actual image, 421 Object image, 422 Background image, 43 Mask image, 431 Random noise, 44 Simulated image, 51 Embedded range information, 511 Cutout X coordinate, 512 Cutout Y coordinate, 513 Object X coordinate, 514 Object Y coordinate, 515 Azimuth angle, 516 Vertical width, 517 Horizontal width, 518 Cutout range, 52 Actual image, 53 Mask image, 531 Random noise, 54 Simulated image, 61 Complementary network, 62 Global discrimination network, 63 Local discrimination network, 64 Patch, 65 Discrimination information, 66 Background-added simulated image, 91 Processor, 92 Memory, 93 Auxiliary storage device, 94 Input / output IF, 95 Communication device, 98 Processing circuit, 99 Signal line.

Claims

1. A learning data generation device that generates a learning dataset for machine learning to detect an object from a remote sensing image, comprising: a style conversion learning unit that performs a first machine learning process using at least one first real image, which is an image showing an image of the object and which is actually captured by remote sensing, at least one first mask image generated by masking an area surrounding the image of the object in each of the at least one first real image, and at least one first simulated image generated by superimposing an image of the object corresponding to each of the at least one first real image on a base image, to generate a first synthetic image showing each of the first simulated image corresponding to each of the at least one first mask image and a background that harmonizes with the unmasked area as an image to be embedded in the masked area in each of the at least one first mask image, and adjusts parameters in the first machine learning process; a second learning data generation unit that generates a second mask image by masking a region indicated by embedding range information indicating a range in which a second simulated image indicating an image of the object is to be embedded in a second actual image, which is an image actually captured by remote sensing, and generates a second composite image indicating each of the second simulated image and a background based on a second machine learning that is equivalent to the first machine learning and that uses the adjusted parameters, the second simulated image, and the second mask image as an image to be embedded in the masked region of the generated second mask image; A learning data generating device comprising:

2. each of the first machine learning algorithm and the second machine learning algorithm is a generative adversarial network; In the first machine learning, the style conversion learning unit a global classification network which is a neural network that receives an input of any one of the at least one first real images or the target-embedded image and outputs a result of classifying whether or not the input is a real image; and a local classification network which is a neural network that receives an input of an image of a region surrounding an image of the object in any one of the at least one first real images or an image of a region surrounding an image of the object in the target-embedded image and outputs a result of classifying whether or not the input is a part of a real image, Adjusting parameters of the complementary network so that the global discriminant network and the local discriminant network tend to output erroneous results; Adjusting parameters of the global identification network and the local identification network so that the accuracy of each of the global identification network and the local identification network is high; The training data generation device according to claim 1 , wherein the second training data generation unit uses a result of adjusting a parameter of the complementation network in the second machine learning.

3. when each of the at least one first simulated image is defined as a first simulated image of a target, and a first real image corresponding to the first simulated image of the target is defined as a first real image of a target, the first simulated image of the target is an image generated by simulating the incidence and reflection of an electromagnetic wave on the target based on an imaging condition corresponding to the first real image of the target and a direction of an image of the target in the first real image of the target, the embedding range information includes information indicating a direction of the object; 3. The learning data generation device according to claim 1 or 2, wherein the second learning data generation unit generates the second simulated image by simulating the incidence and reflection of electromagnetic waves at the object based on the imaging conditions corresponding to the second actual image and the orientation of the object indicated by the embedding range information.

4. The learning data generation device according to claim 3, wherein the orientation of the image of the object in the first real image of the object and the area surrounding the image of the object in the first real image of the object are each determined based on annotation data indicating the orientation of the image of the object in the first real image of the object and the area in which the object exists, respectively.

5. In each of the at least one first mask image, a masked region is a region onto which random noise is superimposed; 3. The training data generating device according to claim 1, wherein in the second mask image, a masked region is a region onto which random noise is superimposed.

6. A learning data generation method executed by a learning data generation device that is a computer that generates a learning dataset for machine learning to detect an object from a remote sensing image, comprising: the learning data generation device performs a first machine learning, which is machine learning using at least one first real image, which is an image containing an image of the object and which is an image actually captured by remote sensing, at least one first mask image generated by masking an area surrounding the image of the object in each of the at least one first real image, and at least one first simulated image generated by superimposing an image of the object corresponding to each of the at least one first real image on a base image, to generate a first synthetic image showing each of the first simulated image corresponding to each of the at least one first mask image and a background that harmonizes with the unmasked area, as an image to be embedded in the masked area in each of the at least one first mask image, thereby adjusting parameters in the first machine learning; The learning data generation method includes: generating a second mask image by masking an area indicated by embedding range information indicating the area in which a second simulated image showing an image of the object is to be embedded in a second actual image, which is an image actually captured by remote sensing; and generating a second composite image indicating each of the second simulated image and a background based on second machine learning, which is machine learning equivalent to the first machine learning, using adjusted parameters, the second simulated image, and the second mask image as an image to be embedded in the masked area of ​​the generated second mask image.

7. A learning data generation program executed by a learning data generation device, which is a computer that generates a learning data set for machine learning to detect an object from a remote sensing image, comprising: a style conversion learning process for adjusting parameters in the first machine learning by performing a first machine learning process using at least one first real image, which is an image showing an image of the object and is actually captured by remote sensing, at least one first mask image generated by masking an area surrounding the image of the object in each of the at least one first real image, and at least one first simulated image generated by superimposing an image of the object corresponding to each of the at least one first real image on a base image, to generate a first synthetic image showing each of the first simulated image corresponding to each of the at least one first mask image and a background that harmonizes with the unmasked area as an image to be embedded in the masked area in each of the at least one first mask image; and a second learning data generation process for generating a second mask image by masking a region indicated by embedding range information indicating a range in which a second simulated image indicating an image of the object is to be embedded in a second actual image, which is an image actually captured by remote sensing, and generating a second composite image indicating each of the second simulated image and a background based on a second machine learning process that is equivalent to the first machine learning process and that uses the adjusted parameters, the second simulated image, and the second mask image as an image to be embedded in the masked region of the generated second mask image; A learning data generation program that causes the learning data generation device to execute the above.

Images