Image generation device and image generation method

Abstract

To provide an image generation device and an image generation method for further simply generating a high-resolution image from a low-resolution image.SOLUTION: An image generation device 1 includes: a first acquisition unit 10 that acquires a target image obtained by enlarging an image I1' including an entire set area A to size of an image I1; and a trained generator G1' constructed using a cycle GAN, for generating an image I2 of a second domain, which is an image obtained by integrating an image i2 of a divided region A1 obtained by dividing the region A into a plurality of regions from the image I1 of a first domain, which is an enlarged image including the entire region A. The image generation device also includes: a generation unit 13 that generates the image I2 by providing the target image to the trained generator G1'; and a presentation unit 15 that presents the generated image I2. The image I1 and the image I2 have the same number of pixels.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image generating device and an image generating method. [Background technology]

[0002] Image processing for increasing the resolution of an image has been known for some time. For example, Patent Document 1 discloses a technology for performing super-resolution processing of an unknown input image using a machine learning model trained using low-resolution images and high-resolution images for training.

[0003] However, in super-resolution processing using the machine learning model described in Patent Document 1, a large number of training images are required to maintain a constant inference accuracy of the machine learning model. Conventionally, super-resolution processing of images taken from the sky by drones, airplanes, etc. has been useful for managing infrastructure, farms, etc. When using the machine learning model described in Patent Document 1 to increase the resolution of images taken from the sky, it can be difficult to prepare a large number of training images, each paired with a low-resolution image and a high-resolution image taken from the sky. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2022-081653 [Non-patent literature]

[0005] [Non-Patent Document 1] Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”, ICCV, pages 2223-2232, 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, with conventional techniques, it has been difficult to generate a high-resolution image from a low-resolution image more easily.

[0007] The present invention has been made to solve the above-mentioned problems, and has an object to generate a high-resolution image from a low-resolution image more easily. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the image generating device of the present invention comprises an acquisition unit configured to acquire a target image obtained by enlarging an image including the entire set area to the size of a first image, and a trained generator constructed using cycle GAN that generates a second image of a second domain, which is an image obtained by integrating images of divided areas obtained by dividing the area into multiple parts from the first image of a first domain obtained by enlarging the image including the entire area, a generation unit configured to provide the target image to the trained generator to generate the second image, and a presentation unit configured to present the generated second image, wherein the first image and the second image have the same number of pixels.

[0009] In addition, the image generating device of the present invention may further include a learning unit configured to use the cycle GAN to learn a generator that generates the second image of the second domain from the first image of the first domain.

[0010] In order to solve the above-mentioned problems, the image generation method of the present invention includes an acquisition step of acquiring a target image obtained by enlarging an image including the entire set area to the size of a first image; a generation step of providing the target image to a trained generator that generates a second image of a second domain, which is an image obtained by integrating images of divided areas obtained by dividing the first image of a first domain, which is an enlarged image including the entire area, constructed using a cycle GAN, to generate the second image; and a presentation step of presenting the generated second image, wherein the first image and the second image have the same number of pixels.

[0011] Furthermore, the image generation method according to the present invention may further include a learning step of using the cycle GAN to train a generator that generates the second image of the second domain from the first image of the first domain. [Effects of the Invention]

[0012] According to the present invention, a trained generator is provided that generates a second image of a second domain, which is an image obtained by dividing a first image of a first domain into a plurality of domains, from an enlarged image including the entire region, constructed using a cycle GAN, and provides a target image to the trained generator to generate the second image. This makes it possible to more easily generate a high-resolution image from a low-resolution image. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image generation system including an image generation device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram for explaining an outline of the image generating device according to this embodiment. [Figure 3] FIG. 3 is a block diagram showing the configuration of a learning unit included in the image generating device according to this embodiment. [Figure 4] FIG. 4 is a diagram illustrating the configuration of a learning unit included in the image generating device according to this embodiment. [Figure 5] FIG. 5 is a diagram illustrating the configuration of a learning unit included in the image generating device according to this embodiment. [Figure 6] FIG. 6 is a block diagram showing the hardware configuration of the image generating device according to this embodiment. [Figure 7] FIG. 7 is a flowchart showing an outline of the operation of the image generating device according to this embodiment. [Figure 8] FIG. 8 is a flowchart showing the learning process performed by the image generating device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to FIGS.

[0015] [Image generation system configuration] First, an overview of an image generation system including an image generation device 1 according to an embodiment of the present invention will be described.

[0016] FIG. 1 is a block diagram showing the configuration of an image generation system including an image generation device 1 according to an embodiment of the present invention. The image generation system according to this embodiment includes the image generation device 1 and a UAV (Unmanned Aircraft Vehicle) 2. The image generation device 1 and the UAV 2 are connected to each other so that they can communicate with each other via a wireless communication network NW such as LTE / 4G, 5G, or 6G. The image generation system according to this embodiment generates a higher-resolution image I2 from an image I1 (target image) that includes the entire area A captured by the UAV 2. The area A is, for example, farmland measuring 100 × 100 m, and the growth of crops cultivated in the area A is managed using the high-resolution image I2.

[0017] The UAV2 is an unmanned aerial vehicle such as a drone, capable of remotely controlling flight. The UAV2 includes a processor, a main memory device, an auxiliary memory device, a communication interface, etc. The UAV2 also includes a gyro sensor, an acceleration sensor, a flight controller, a GPS receiver, a camera, etc.

[0018] The ground sampling distance (GSD) of images captured using UAV2's camera is determined by the shooting altitude and the camera's sensor size. UAV2 can capture an area from the sky that corresponds to the shooting altitude and the camera's field of view. For simplicity's sake, the camera's performance is assumed to be 1 million pixels, capturing images with an aspect ratio of 1000 x 1000 pixels.

[0019] As shown in FIG. 1, UAV2 captures the entire area A set as the capture range as a single image I1'. GPS coordinate information is attached to image I1'. For example, coordinate information may be included for each image I1' or for each pixel. UAV2 transmits the captured image I1' of area A to image generation device 1 via wireless communication network NW. In addition to image I1' containing the entire area A, UAV2 can capture image I1' and images i2 of multiple divided areas A1 obtained by dividing area A as training data used in learning by learning unit 12, which will be described later. GPS coordinate information is attached to each image i2.

[0020] [Functional block of image generation device] As shown in Fig. 1, the image generation device 1 includes a first acquisition unit 10, a second acquisition unit 11, a learning unit 12, a generation unit 13, a memory unit 14, and a presentation unit 15. The image generation device 1 generates and presents an image I2 by integrating images i2 of divided regions A1, which are obtained by dividing region A into multiple regions, from a single low-resolution image I1' containing the entire region A and transmitted from a UAV2, using a trained generator G1' constructed in advance using cycle GAN.

[0021] The first acquisition unit 10 (acquisition unit) acquires a target image obtained by enlarging an image I1' including the entire set area A to the size of the image I1 (first image). More specifically, the first acquisition unit 10 acquires the image I1' of the area A photographed by the UAV 2 via the wireless communication network NW. The first acquisition unit 10 generates an image I1 by simply enlarging the scale of the image I1'. By enlarging the image I1', an image I1 with an increased number of pixels is acquired. For example, an image I1 obtained by enlarging an image I1' with a pixel count of 1000 x 1000 [px] by 100 x 100 times is an image I1 in which there are 100 x 100 pixels whose RGB components are the same as one pixel of the image I1'. The image I1 acquired by the first acquisition unit 10 is an image to be converted into a high-resolution image.

[0022] The second acquisition unit 11 acquires training data images for learning processing using cyclic GAN by the learning unit 12. The second acquisition unit 11 acquires, as training data, an image I1 (first image) obtained by enlarging an image I1' of area A previously photographed by the UAV 2, and an image I2 (second image) obtained by integrating images i2 of divided areas A1 obtained by dividing area A into multiple areas. The second acquisition unit 11 acquires the image I1' of the entire area A photographed by the UAV 2 and the image i2 of divided area A1 via the wireless communication network NW, and further performs an enlargement process on image I1' and integrates image i2 to obtain images I1 and I2 of the training data.

[0023] The learning unit 12 uses a cycle GAN to learn a generator G1 that generates an image I2 of a second domain from an image I1 of a first domain.

[0024] Fig. 2 is a diagram for explaining an overview of the image generating device 1 according to this embodiment. Fig. 2(a) is a diagram schematically showing an image I1, and Fig. 2(b) is a diagram schematically showing an image I2. Fig. 2(c) is a diagram showing the image I1 and the image I2 captured by the UAV2.

[0025] As shown in (a) to (c) of Figure 2, image I1' is an image captured by UAV2 of the entire area A as a single image, and image I1 is an enlarged image of image I1'. On the other hand, image I2 is an image obtained by integrating images i2, which are images captured by UAV2 of multiple divided areas A1 into which area A is divided. Image i2 is an image captured by UAV2 at a lower altitude than when image I1' was captured, and has a ground resolution that is lower than the ground resolution of image I1'.

[0026] As mentioned above, area A is farmland with an area of ​​100 x 100 m. The grid-like area of ​​image I1' shown in Figure 2(a) schematically represents one pixel. Also, as mentioned above, image I1' before enlargement is an image of 1,000 x 1,000 px. In image I1, which is enlarged 100 x 100 times to (1,000 x 100) x (1,000 x 100) px, the information in the original image I1' is diffused, and the details between the pixels are lost, causing the crops to appear blurry.

[0027] On the other hand, image I2 shown in Figure 2(b) is an image i2 of 1,000 x 1,000 pixels taken by UAV2 of each of multiple divided areas A1, each divided into 1 x 1 meters. In this case, UAV2 takes 10,000 images i2. As indicated by the arrow in Figure 2(b), by combining the 1,000 x 1,000 pixels of images i2, a (1,000 x 100) x (1,000 x 100) pixel image I2 covering the entire area A is obtained. Image I2, which combines images i2, has a sufficiently low ground resolution, allowing the crops to be clearly visible. Thus, the resolution of image I2, which combines images i2 of divided area A1, is higher than the resolution of image I1, which is an enlarged image I1' that covers the entire area A. Furthermore, the size of image I1, which is an enlarged image I1', is the same as the size of image I2, which combines images i2 of divided area A1.

[0028] The learning unit 12 performs training of the cycle GAN using images I1 and I2, which are enlarged versions of image I1' shown in (a) and (b) of Figures 2, as training data. As shown in Figure 3, the learning unit 12 performs training of the cycle GAN, which has two generators G1 and G2 and two classifiers D1 and D2. The cycle GAN is a type of GAN (Generative Adversarial Network) and is a generative model that can perform image conversion between different domains (see Non-Patent Document 1).

[0029] While GAN is a model with one generator and one classifier, Cycle GAN is composed of two generators G1 and G2 and two classifiers D1 and D2, with one generator and one classifier for each domain. That is, for the first domain, the generator G1 and the classifier D2 are provided, and for the second domain, the generator G2 and the classifier D1 are provided.

[0030] The learning unit 12 uses a cycle GAN to train a generator G1 that generates an image I2 of a second domain from an image I1 of a first domain. A domain refers to a collection of data with certain characteristics. Image I1, which is an enlarged image I1' that includes the entire area A, and image I2, which is an integrated image i2 of divided area A1, have different characteristics and are images of different domains with no pixel-by-pixel correspondence.

[0031] The cycle GAN includes a pair of a generator G1 that converts an image I1 of a first domain into an image I2 of a second domain, and a classifier D1 that determines whether the image I2 converted by the generator G1 is true or false, and a pair of a generator G2 that inversely converts the image I2 of the second domain into the image I1 of the first domain, and a classifier D2 that determines whether the inversely converted image I1 is true or false. In this way, the cycle GAN has a cyclic structure of conversion and inverse conversion using two pairs of generators G1 and G2 and classifiers D1 and D2 to learn the relationship between the first domain and the second domain.

[0032] 4 and 5 are diagrams schematically illustrating the neural network configurations of generators G1 and G2 and discriminators D1 and D2 of the cyclic GAN used by the learning unit 12. The generators G1 and G2 are configured as neural networks having an input layer, a hidden layer, and an output layer. CNN or ResNet can be used as the neural networks configuring the generators G1 and G2.

[0033] FIG. 4 is a diagram showing a schematic diagram of the neural network configuration of the generator G1. As shown in FIG. 4, the input nodes of the generator G1 receive pixel values ​​(x1 to x n ) is input. The small squares shown on the input layer side of Figure 4 represent 1x1 [px] pixels. The generator G1 performs a product-sum operation on the input and weight parameters and threshold processing using the activation function to generate output data G1(x1) to G1(x n ) is output. Output data G1(x1) to G1(x n ) is a matrix with 10,000 elements corresponding to 100 x 100 [px], and is represented diagrammatically by the rectangle on the output layer side in Figure 4. The output data from generator G1 is image I2 of the second domain, converted from image I1 of the first domain. Similarly, generator G2 receives the pixel values ​​of each image i2 that make up image I2 of the second domain, and then performs product-sum operations on the input and weight parameters and threshold processing using an activation function to obtain output data of the first domain.

[0034] Similarly, the classifiers D1 and D2 are also configured with neural networks having an input layer, a hidden layer, and an output layer. A CNN can be used as the neural network that configures the classifiers D1 and D2. Figure 5 is a diagram that schematically illustrates the neural network configuration of the classifier D1. The input node of the classifier D1 is given pixel values ​​of the image I2. In the example of Figure 5, output data G1(x1) to G1(x n ) is given to the input node of the discriminator D1. The output data G1(x1) to G1(x n ) is the pixel value of image I2 generated by generator G1.

[0035] Classifier D1 performs a product-sum operation on the input and weight parameters and threshold processing using an activation function, and then outputs a binary output of 1 or 0. An output value of 1 indicates that image I2 is a genuine image, and an output value of 0 indicates that input image I2 is a pseudo image.

[0036] Similarly, for the first domain classifier D2, the value of each pixel constituting the first domain image I1 is input to the input node, and the neural network outputs a value of 1 or 0 as its calculation result. An output value of 1 indicates that the input image I1 is a genuine image, and an output value of 0 indicates that the input image I1 is a pseudo image.

[0037] Here, the learning unit 12 introduces an objective function that combines an adversarial loss and a cycle consistency loss to learn the relationship between the first domain and the second domain using cycle GAN. Similar to a conventional GAN, an adversarial loss is used in learning the transformation of an image from the first domain to the second domain and the inverse transformation of an image from the second domain to the first domain. In addition to this adversarial loss, the learning unit 12 incorporates a cycle consistency loss into the objective function to learn a relationship in which a generator G1 that performs image transformation from the first domain to the second domain and a generator G2 that performs inverse transformation from the second domain to the first domain maintain consistency in the cycle of repeated transformation and inverse transformation.

[0038] The adversarial loss used by the learning unit 12 will be explained below by taking as an example a network consisting of a generator G2 that converts images from the second domain to the first domain and a classifier D2 that judges the authenticity of the converted image. First, let us denote the data of the real image of image I1 in the first domain as x, the predicted value output by classifier D2 as y, and the correct label as t. The correct label t is set to 1 for the real image of image I1 in the first domain, and 0 for the pseudo image of image I1 generated by generator G2. In this case, classifier D2 calculates the cross entropy E in the following equation (1) for a binary classification problem: CE It can be expressed as:

[0039]

number

[0040] The first term in the brace of the above equation (1) represents t n lny n In this case, the predicted value y n However, the correct label of the real image of image I1 is t n = 1. On the other hand, the second term in the braces represents (1-t n )ln(1-y n ), the predicted value y n is the correct label value (1-t n ) = 0. In this way, the cross entropy E CE is the maximum value when the predicted value matches the correct label value.

[0041] Here, the generators G1 and G2 (hereinafter collectively referred to as “generator G”) that make up the cycle GAN are G ,θ G and the function G(w G ,θ G ) and the classifiers D1 and D2 (hereinafter collectively referred to as "classifier D") are expressed as follows: D ,θ D and function D(w D ,θ D ) The cross entropy E in the above equation (1) CE The adversarial loss L of the model with a generator G and a discriminator D based on GAN can be expressed by the following equation (2).

number

[0042] The first term of the above equation (2) represents E D(x)=1 lnD(w D ,θ D ) is the expected value that the classifier D will classify a real image as real. D(x)=0ln(1-D(G(w G ,θ G ),w D ,θ D )) is the expected value that the classifier D will classify the pseudo-image generated by the generator G as a pseudo-image. In the adversarial learning of cycle GAN, the adversarial loss L for each pair of the generator G and the classifier D is GAN The generator G and the classifier D are adversarially trained using min-max optimization. Therefore, the generator G is trained to generate pseudo-images that can fool the classifier D, and the classifier D is trained to identify the pseudo-images generated by the generator G as pseudo-images.

[0043] In the adversarial learning of the classifier D, when a real image is given, the classifier D outputs an output close to y=1, and the adversarial loss L in the above equation (2) is GAN On the other hand, when a pseudo-image is given, the adversarial loss L GAN Learning is performed to maximize the second term of

[0044] In the adversarial learning of the generator G, D(G(w G ,θ G ),w D ,θ D ) is close to 1. G ,θ G ) to obtain the adversarial loss L GAN The learning unit 12 uses a learning procedure in which the parameters of the generator G and the parameters of the discriminator D are updated alternately.

[0045] Here, as shown in Figure 3, the generator G1(w G1 ,θ G1 ) is input with the real image x of image I1, and the converted output is denoted as G1(x), and the generator G2(w G2 ,θ G2 ) is input with the real image y of image I2, and the converted output is represented as G2(y). D1 ,θ D1 ) when the real image y of image I2 is input, the output of the authenticity judgment is represented as D1(y).

[0046] Furthermore, the discriminator D1(w D1 ,θ D1 ), generator G1(w G1 ,θ G1 ) is input, the output of the true / false judgment when the output G1(x) which is a pseudo image of the image I2 generated by the classifier D2(w D2 ,θ D2 ) when the real image x of image I1 is input, the output of the authenticity judgment is represented as D2(x), and the classifier D2(w D2 ,θ D2 ), generator G2(w G2 ,θ G2 ) is input, the output of the true / false judgment when the output G2(y) which is a pseudo image of the image I1 generated in the above is input is represented as D2(G2(y)).

[0047] From these definitions, the adversarial loss L in Equation (2) above is GAN is the adversarial loss L in the following equation (3). GAN1 , and the adversarial loss L in the following equation (4) GAN2 It is expressed as:

[0048]

number

[0049] Using the above equation (3), the image I1 in the first domain is transformed into the image I2 in the second domain by the generator G1, and the adversarial loss L of the network is used to judge the authenticity of the transformed image I2 by the discriminator D1. GAN1 In addition, the adversarial loss L of the network that converts the image I2 of the second domain into the image I1 of the first domain using the generator G2 and judges the authenticity of the converted image I1 using the classifier D2 is calculated using the above formula (4). GAN2 is calculated.

[0050] The learning unit 12 calculates the adversarial loss L of the network related to the generator G1 and the discriminator D1 in the above formula (3). GAN1 , and the adversarial loss L of the network related to the generator G2 and the discriminator D2 in the above equation (4). GAN2 Optimize each of them.

[0051] The learning unit 12 further calculates the cycle consistency loss L cyc The input image is transformed by generator G using the generator G, and the transformed image is then inversely transformed by another generator G to reconstruct the original image, which is then trained to match the input image before transformation.

[0052]

number

[0053] The cycle consistency loss L in equation (5) above cyc is the norm L 1 The loss is expressed in the form of an expectation using the formula (5), and the loss is calculated by comparing the image restored by inversely transforming an image transformed by generator G with the original input image, pixel by pixel. The first term in the formula (5) evaluates whether the original image x of image I1 is input as an image and transformed by generators G1 and G2 in that order to return to image I1, based on the output G2(G1(x)) after transformation and the absolute value of x. The loss in this first term is called the cycle consistency loss L cyc1 The second term uses the real image y of image I2 as the input image and applies transformations to generators G2 and G1 in that order, and evaluates whether it returns to image I2 based on the transformed output G1(G2(y)) and the absolute value of y. The loss of the second term is called the cycle consistency loss L cyc2 That's what they say.

[0054] The learning unit 12 calculates the adversarial loss L GAN1 , L GAN2 and the cycle consistency loss L in equation (5) above. cyc By combining these, we set the final objective function L of the cycle GAN, which is expressed as the following equation (6).

number

[0055] Returning to Figure 1, the generation unit 13 includes a trained generator G1' that generates an image I2 (second image) of a second domain, which is an image obtained by integrating images i2 of divided regions A1 into which the region A is divided, from an image I1 (first image) of a first domain, which is an enlarged image of a single image I1' including the entire region A, constructed using cycle GAN, and provides the trained generator G1' with the target image I1 to generate the image I2.

[0056] The storage unit 14 stores the trained generator G1′. The storage unit 14 can also store the pre-training generators G1 and G2 and the discriminators D1 and D2 that constitute the cycle GAN.

[0057] The presentation unit 15 presents the image I2 generated by the generation unit 13. The presentation unit 15 can transmit the image I2 generated as a high-resolution image to an external terminal device such as a farm manager via, for example, a wireless communication network NW.

[0058] [Hardware configuration of image generation device] Next, an example of a hardware configuration for realizing the image generating device 1 having the above-described functions will be described with reference to FIG.

[0059] 6, image generating device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface 104, an auxiliary memory device 105, and an input / output (I / O) 106, all of which are connected via a bus 101, and a program that controls these hardware resources. Image generating device 1 can also include a display device 107 connected via bus 101.

[0060] The processor 102 is realized by a CPU, a GPU, an FPGA, an ASIC, or the like.

[0061] The main memory device 103 pre-stores programs for the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 implement the functions of the image generating device 1, such as the first acquisition unit 10, the second acquisition unit 11, the learning unit 12, the generation unit 13, and the presentation unit 15 shown in FIG.

[0062] The communication interface 104 is an interface circuit for connecting the image generating device 1 to various external electronic devices via a network.

[0063] The auxiliary storage device 105 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The auxiliary storage device 105 can use a semiconductor memory such as a hard disk or flash memory as the storage medium.

[0064] The auxiliary storage device 105 has a program storage area for storing the cycle GAN learning program and image generation program executed by the image generating device 1. The auxiliary storage device 105 realizes the storage unit 14 described in FIG. 1. Furthermore, for example, the auxiliary storage device 105 may have a backup area for backing up the above-mentioned data, programs, etc.

[0065] The input / output I / O 106 is an input / output device that inputs signals from external devices and outputs signals to external devices.

[0066] The display device 107 is configured by an organic EL display, a liquid crystal display, etc. The display device 107 can also realize the presentation unit 15.

[0067] [Image generation device operation] Next, the operation of the image generating device 1 having the above-described configuration will be described with reference to the flowcharts of FIGS.

[0068] 7 is a flowchart showing the operation of the image generating device 1. First, the learning unit 12 of the image generating device 1 uses a cycle GAN to learn a generator G1 that generates an image I2 of a second domain from an image I1 of a first domain (step S1). Details of the learning process in step S1 will be described later.

[0069] Thereafter, the first acquisition unit 10 acquires, as a target image, an image I1 that is an enlarged version of an image I1' that includes the entire area A and was captured by the UAV2 (step S2). Next, the generation unit 13 provides the image I1 acquired in step S2 to the trained generator G1' to generate an image I2 (step S3). The image I1 and the generated image I2 have the same number of pixels. Thereafter, the presentation unit 15 presents the image I2 generated in step S3 (step S4). In step S4, the presentation unit 15 can transmit the image I2, which is a high-resolution image, to a terminal device or the like external to the farmland manager.

[0070] Next, the learning process performed by the image generating device 1 will be described with reference to the flowchart of FIG. 7 and the block diagram of the learning unit 12 of FIG.

[0071] First, the second acquisition unit 11 acquires training data (step S100). The second acquisition unit 11 acquires a real image x of an image I1 captured by the UAV2. real , and the real image y of image I2 real In this embodiment, the second acquisition unit 11 acquires, as training data, a plurality of images I1 based on images I1' captured by the UAV 2 of the area A at different angles and at different times. real For the training data of (1), multiple images I2 taken at different angles and different times are acquired as training data. These training data are set as “input x” and “input y” in Figure 3, respectively.

[0072] Next, the learning unit 12 inputs training data to two classifiers D1 and D2 constituting the cycle GAN, respectively, to have them learn real images, and calculates the parameter w D1 ,θD1 , and the parameter w of the discriminator D2 D2 ,θ D2 In step S101, a plurality of real images y of an image I1' that is an enlarged image I1' including the entire area A photographed at different angles and different times are updated. real is input to the classifier D1, and the parameter w is calculated using the backpropagation method or the like so that the tolerance entropy error in the above formula (1) is minimized. D1 ,θ D1 Similarly, for the classifier D2, the real image x of the multiple images I2 obtained by integrating the images i2 of the divided area A1 taken at different angles or different times is updated. real is given as input, and the parameter w is calculated using backpropagation or other methods to minimize the tolerance entropy error in equation (1) above. D2 ,θ D2 Update.

[0073] Next, the learning unit 12 performs adversarial learning on a network consisting of a pair of the generator G1 and the classifier D1, and on a network consisting of a pair of the generator G2 and the classifier D2, in the following steps.

[0074] First, the learning unit 12 calculates the adversarial loss L GAN1 In more detail, the learning unit 12 trains the classifier D1 using the real image y of the image I2 obtained by integrating the image i2 of the divided region A1, as shown in FIG. real is given as input, and the adversarial loss L GAN1 The gradient of dL GAN1 / dw D1 ,dL GAN1 / dθ D1 Calculate the adversarial loss L by backpropagation or other methods. GAN1 parameter w D1 ,θ D1 Furthermore, the learning unit 12 updates the pseudo-image y of the image I2 generated by the generator G1 in the classifier D1. fake is given as input, and similarly the adversarial loss L GAN1 parameter w D1 ,θ D1Note that the learning of the classifier D1 is performed by updating the parameter w G1 ,θ G1 is fixed.

[0075] Next, the learning unit 12 calculates the adversarial loss L GAN1 In more detail, the learning unit 12 trains the generator G1 using the real image x of the image I1, which is an enlarged image I1' including the entire region A, as shown in FIG. real is given as input, and the adversarial loss L GAN1 Gradient of -dL GAN1 / dw G1 ,-dL GAN1 / dθ G1 Calculate the adversarial loss L by backpropagation or other methods. GAN1 parameter w so that is minimized. G1 ,θ G1 The learning of the generator G1 is performed by updating the parameter w D1 ,θ D1 is fixed.

[0076] Next, the learning unit 12 calculates the adversarial loss L GAN2 In more detail, as shown in FIG. 3, the learning unit 12 trains the classifier D2 by using the real image x of the image I1. real is given as input, and the adversarial loss L GAN2 The gradient of dL GAN2 / dw D2 ,dL GAN2 / dθ D2 Calculate the adversarial loss L by backpropagation or other methods. GAN2 parameter w D2 ,θ D2 Furthermore, the learning unit 12 generates a pseudo-image y of the image I1, which is an enlarged version of the image I1' of the entire region A generated by the generator G2, in the classifier D2. fake is given as input, and similarly the adversarial loss L GAN2 parameter w D2 ,θ D2 Note that the learning of the classifier D2 is performed by updating the parameter wG2 ,θ G2 is fixed.

[0077] Next, the learning unit 12 calculates the adversarial loss L GAN2 In more detail, the learning unit 12 trains the generator G2 using the real image y of the image I2 obtained by integrating the image i2 of the divided region A1, as shown in FIG. real is given as input, and the adversarial loss L GAN2 Gradient of -dL GAN2 / dw G2 ,-dL GAN2 / dθ G2 Calculate the adversarial loss L by backpropagation or other methods. GAN2 parameter w so that is minimized. G2 ,θ G2 The learning of the generator G2 is performed by updating the parameter w D2 ,θ D2 The learning unit 12 performs the learning in steps S104 and S105 to obtain the adversarial loss L GAN2 Ask for.

[0078] Next, the learning unit 12 calculates the cycle consistency loss L cyc is calculated in the following steps. First, the learning unit 12 calculates the real image x of the image I1 of the entire region A. real and the cycle consistency loss L between the reconstructed image I1' of the entire region A and the pseudo-image G2(G1(x)) of the enlarged image I1. cyc1 is calculated and the generators G1 and G2 are trained (step S106).

[0079] More specifically, as shown in FIG. 3, the learning unit 12 enlarges the image I1' of the entire region A to obtain a genuine image x real is given to generator G1, and the generated pseudo image G1(x) of image I2 obtained by integrating image i2 of divided area A1 is further given as input to generator G2. The pseudo image G2(G1(x)) of image I1 obtained by enlarging image I1' of the entire area A further reconstructed by generator G2, and the real image x of image I1 obtained by enlarging image I1' of the entire area A are given as input. realCycle consistency loss with L cyc1 The learning unit 12 calculates the original image, the real image x real and the reconstructed pseudo image G2(G1(x)) are compared pixel by pixel, and the distance L 1 By minimizing the cycle consistency loss L cyc1 Minimize.

[0080] Next, the learning unit 12 calculates the real image y of the image I2 obtained by integrating the image i2 of the divided area A1. real and the cycle consistency loss L between the pseudo-image G1(G2(y)) of the image I2 obtained by integrating the image i2 of the reconstructed segmented area A1. cyc2 , and trains the generators G1 and G2 (step S107). More specifically, as shown in FIG. 3, the training unit 12 calculates the real image y real is given to generator G2, and the generated pseudo image G2(y) of image I1 of the entire region A is further given as input to generator G1. The pseudo image G1(G2(y)) of image I2 obtained by integrating image i2 of divided region A1, which is further reconstructed by generator G1, and the real image y of image I2 obtained by integrating image i2 of divided region A1 are given as input to generator G1. real Cycle consistency loss with L cyc2 The learning unit 12 calculates the original image, the real image y real and the reconstructed pseudo image G1(G2(y)) are compared pixel by pixel, and the distance L 1 By minimizing the cycle consistency loss L cyc2 Minimize.

[0081] Next, the learning unit 12 calculates the adversarial loss L calculated in steps S102 to S107. GAN1 , L GAN2 , and cycle consistency loss L cycThen, the objective function L of the above equation (6) is set, and the processes from step S102 to step S107 are repeated until the objective function L converges (step S108: NO). The learning unit 12 stores the generator G1 obtained when the objective function L converges as a trained generator G1' in the storage unit 14 (step S109). Through the above learning process, a trained generator G1' is constructed that converts the image I1 of the entire area A in the first domain into an image I2 obtained by integrating the image i2 of the divided area A1 in the second domain.

[0082] As described above, according to the image generating device 1 of this embodiment, a generator G1 is constructed using cycle GAN to convert an image I1 obtained by enlarging an image I1' of the entire area A in the first domain into an image I2 obtained by integrating an image i2 of the divided area A1 in the second domain, thereby making it easier to generate a high-resolution image from a low-resolution image.

[0083] Furthermore, according to the image generating device 1 of this embodiment, the trained generator G1' constructed using cycle GAN is used to generate image I2 by integrating images i2 of the divided region A1. Therefore, even if it is difficult to obtain a large number of images I1 and I2 as training data, the generator G1 can be trained.

[0084] In the above-described embodiment, the learning unit 12 uses the adversarial loss L GAN , and cycle consistency loss L cyc However, the learning unit 12 sets the objective function L by combining the adversarial loss L GAN , and cycle consistency loss L cyc In addition, an objective function L having a identity loss may be set. Identity loss is, for example, calculating a loss for each pixel so that the input image to the generator G matches the pseudo image generated by the generator G. By further introducing identity loss, the generator G1 is trained to generate image I2 by transforming only the necessary parts of the input image I1 of the entire region A.

[0085] The above describes embodiments of the image generation device and image generation method of the present invention, but the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can make within the scope of the invention described in the claims are possible. [Explanation of symbols]

[0086] 1...image generation device, 10...first acquisition unit, 11...second acquisition unit, 12...learning unit, 13...generation unit, 14...memory unit, 15...presentation unit, 2...UAV, 101...bus, 102...processor, 103...main memory device, 104...communication interface, 105...auxiliary memory device, 106...input / output I / O, 107...display device, G, G1, G2...generator, D, D1, D2...classifier, G1'...trained generator, NW...wireless communication network.

Claims

1. an acquisition unit configured to acquire a target image obtained by enlarging an image including the entire set area to the size of the first image; A generator includes a trained generator that generates a second image of a second domain, which is an image obtained by integrating images of divided regions obtained by dividing a first image of a first domain into a plurality of regions, from the first image of a first domain constructed using a cycle GAN, which is an enlarged image including the entire region, and is configured to provide the target image to the trained generator to generate the second image; a presentation unit configured to present the generated second image; Equipped with The first image and the second image have the same number of pixels. An image generating device characterized by:

2. 2. The image generating device according to claim 1, and a learning unit configured to use the cycle GAN to learn a generator that generates the second image of the second domain from the first image of the first domain. An image generating device characterized by:

3. an acquisition step of acquiring a target image by enlarging an image including the entire set area to the size of the first image; a generation step of providing the target image to a trained generator that generates a second image of a second domain, the second image being an image obtained by integrating images of divided regions obtained by dividing a first image of a first domain into a plurality of regions, the second image being constructed using a cycle GAN and including an enlarged image including the entire region; and a presentation step of presenting the generated second image; Equipped with The first image and the second image have the same number of pixels. An image generating method comprising:

4. 4. The image generating method according to claim 3, Further, a learning step is provided for learning a generator that generates the second image of the second domain from the first image of the first domain using the cycle GAN. An image generating method comprising:

Images