Image generation device, image generation method, and program

Abstract

To solve the problem that an annotation error may occur in a SAR image.SOLUTION: An image generation device includes: image acquisition means that acquires an optical image in which an object is imaged; annotation data acquisition means that acquires annotation data regarding the object, applied to the optical image; generation means that generates a pseudo SAR image that simulates a SAR image from the optical image by conversion processing; and addition means that adds the pseudo SAR image to training data in association with the annotation data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device, an image generation method, and a program. [Background technology]

[0002] When creating training data for SAR (Synthetic Aperture Radar) images, annotation is performed on the objects captured in the SAR images.

[0003] Patent Document 1 describes reading annotation data that expresses the range of an object captured in a SAR image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-163692 Summary of the Invention [Problem to be solved by the invention]

[0005] When annotating SAR images, it can be difficult to perform annotation accurately, which can lead to annotation errors in SAR images.

[0006] One of the objectives of the present disclosure is to provide an image generation device and the like that can reduce annotation errors when creating training data for SAR images. [Means for solving the problem]

[0007] An image generating device according to one aspect of the present disclosure includes an image acquisition means for acquiring an optical image of an object, an annotation data acquisition means for acquiring annotation data about the object that is added to the optical image, a generation means for generating a pseudo-SAR image that simulates an SAR image from the optical image through a conversion process, and an addition means for linking the pseudo-SAR image to the annotation data and adding it to training data.

[0008] An image generation method according to one aspect of the present disclosure acquires an optical image of an object, acquires annotation data about the object that has been added to the optical image, generates a pseudo-SAR image that simulates an SAR image from the optical image through a conversion process, and links the pseudo-SAR image to the annotation data and adds it to training data.

[0009] A program in one aspect of the present disclosure causes a computer to perform the following processes: acquire an optical image of an object; acquire annotation data about the object that has been added to the optical image; generate a pseudo-SAR image that simulates an SAR image from the optical image through a conversion process; link the pseudo-SAR image to the annotation data; and add the pseudo-SAR image to training data.

[0010] Each program may be stored in a non-transitory computer-readable recording medium. [Effects of the Invention]

[0011] One example of the effect of the present disclosure is that it becomes possible to reduce annotation errors when creating training data for SAR images. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example of a configuration of an image generating device. [Figure 2] FIG. 10 is a diagram showing an example of a flow in which training data of a pseudo SAR image is added. [Figure 3]10 is a flowchart showing the operation of the image generating device. [Figure 4] FIG. 10 is a diagram showing an example of a flow in which training data of an SAR image is added. [Figure 5] FIG. 1 is a block diagram illustrating an example of a configuration of an image generating device. [Figure 6] FIG. 10 is a diagram showing an example of a screen displaying a pseudo SAR image. [Figure 7] FIG. 10 is a diagram showing an example of a screen that displays an optical image and a pseudo SAR image generated from the optical image in association with each other. [Figure 8] FIG. 10 is a diagram showing an example of a screen displaying an optical image and a pseudo-optical image generated based on the optical image. [Figure 9] 10 is a flowchart showing the operation of the image generating device. [Figure 10] FIG. 2 illustrates an example of a hardware configuration of an image generating apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0013] Embodiments of the present disclosure will be described in detail with reference to the drawings.

[0014] [First embodiment] The configuration of the image generating device 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the image generating device 10. The image generating device 10 includes an image acquiring unit 101, an annotation data acquiring unit 102, a generating unit 103, and an adding unit 104.

[0015] The image acquisition unit 101 is one aspect of image acquisition means that acquires an optical image of an object. The optical image is, for example, an image captured using an image sensor that can observe visible light. The optical image is not limited to this. The optical image may also be an image captured using an image sensor that can observe near-infrared light. Furthermore, the optical image is, for example, an image of the earth's surface captured from a satellite or an aircraft. The optical image captures an object. Examples of the object include a ship, an aircraft, or a vehicle. The object is not limited to these. The object may be any object that can be observed using an image.

[0016] The captured optical images are stored in, for example, a database (not shown). The image acquisition unit 101 can acquire the optical images from the database. Here, the image generation device 10 and the database are connected via a wired or wireless network. The image acquisition unit 101 may also acquire optical images captured by a sensor from the sensor. The image generation device 10 and the sensor are connected via a wired or wireless network. The image acquisition unit 101 may acquire one optical image or multiple optical images.

[0017] The annotation data acquisition unit 102 is one aspect of annotation data acquisition means that acquires annotation data about an object that is added to an optical image.

[0018] The annotation data includes information about the object captured in the optical image. The annotation data is assigned to the optical image. The information about the object includes information about the type of object included in the optical image and information about the position of the object included in the optical image. In other words, the annotation data is information that indicates what kind of object is captured and where in the optical image.

[0019] The annotation data acquisition unit 102 acquires annotation data that is assigned to an optical image and relates to an object captured in the optical image. The annotation data acquisition unit 102 can acquire the annotation data by, for example, annotation using a trained model. The annotation data acquisition unit 102 annotates the optical image acquired by the image acquisition unit 101 using the trained model. The annotation data acquisition unit 102 can acquire the annotation data output by the trained model.

[0020] Annotation using the trained model may be performed in an information processing device (not shown) different from the image generating device 10. In this case, the annotation data acquiring unit 102 transmits the optical image acquired by the image acquiring unit 101 to the information processing device. The information processing device performs annotation using the trained model. Then, the annotation data acquiring unit 102 can acquire the annotation data output by the trained model from the information processing device.

[0021] The optical image and the annotation data output from the trained model for the optical image are linked. The trained model may be a publicly known model. The trained model may also be a model that has been additionally trained to detect objects captured in the image.

[0022] The annotation data acquisition unit 102 may acquire, as annotation data, the results of annotation performed by a user on an optical image. The user observes the optical image acquired by the image acquisition unit 101 and makes annotations on the object captured in the optical image. Then, the user inputs the results of the annotation using, for example, an input device (not shown). The annotation data acquisition unit 102 acquires the annotation data input by the user. When the user makes annotations, the optical image and the annotation data for the optical image are linked.

[0023] The annotation data input by the user may be stored in, for example, a database. In this case, the annotation data acquisition unit 102 acquires the annotation data from the database. Here, the optical image and the annotation data for the optical image are associated with each other and stored in the database.

[0024] The method of acquiring annotation data by the annotation data acquisition unit 102 is not limited to these. Annotation of objects included in optical images is performed using known techniques.

[0025] The annotation data acquisition unit 102 may acquire annotation data using, for example, information that assists annotation. Using the information that assists annotation can improve the accuracy of annotation. For example, when the annotation data acquisition unit 102 acquires annotation data through annotation using a trained model, the information that assists annotation may be input to the trained model. Furthermore, when the annotation data acquisition unit 102 acquires annotation data through annotation by a user, the user can perform annotation by referring to the information that assists annotation.

[0026] An example of information that assists annotation is information from an Automatic Identification System (AIS) when the target object is a ship. When the target object is a ship, the annotation data acquisition unit 102 can acquire annotation data that uses AIS information. The AIS information may include the location and type of the ship. Therefore, when the target object is a ship, it is possible to improve the accuracy of the annotation.

[0027] The generation unit 103 is one aspect of a generation means that generates a pseudo SAR image that simulates an SAR image from an optical image through conversion processing. The pseudo SAR image is an image generated based on an optical image. The pseudo SAR image is an image generated through conversion processing so as to look similar to an SAR image.

[0028] The conversion process from an optical image to a pseudo SAR image uses, for example, a GAN (Generative Adversarial Network). The generation unit 103 can use an existing GAN to generate a pseudo SAR image that simulates the optical image from the optical image. The GAN may be one that has undergone additional learning for generating pseudo SAR images. The generation unit 103 uses the GAN to generate one pseudo SAR image from one optical image. The generation unit 103 may also use the GAN to generate multiple pseudo SAR images from one optical image. The method of the conversion process from an optical image to a pseudo SAR image is not limited to this. The conversion process from an optical image to a pseudo SAR image is performed using a known technique.

[0029] Here, the optical image and the pseudo SAR image generated from the optical image are linked. Also, as described above, the optical image and the annotation data for the optical image are linked. Therefore, the pseudo SAR image and the annotation data for the pseudo SAR image are linked. In other words, the annotation data for the optical image and the annotation data for the pseudo SAR image generated from the optical image are the same.

[0030] The generation unit 103 may generate a pseudo optical image that simulates the optical image from the pseudo SAR image through a conversion process. In other words, the generation unit 103 may generate a pseudo optical image that looks similar to the optical image from the pseudo SAR image. Generating a pseudo optical image makes it possible to increase the amount of training data. The generation unit 103 can generate a pseudo optical image from the pseudo SAR image using a GAN. The generation unit 103 can generate a pseudo optical image from the pseudo SAR image using an existing GAN. The GAN may be one that has undergone additional learning to generate pseudo optical images. The method of conversion process from a pseudo SAR image to a pseudo optical image is not limited to this. The conversion process from a pseudo SAR image to a pseudo optical image is performed using a known technique.

[0031] The adding unit 104 is one aspect of adding means that links a pseudo SAR image to annotation data and adds it to training data. The adding unit 104 adds the pseudo SAR image to the training data. At this time, the adding unit 104 links the pseudo SAR image to the annotation data corresponding to the pseudo SAR image and adds it to the training data. As described above, the pseudo SAR image generated by the generating unit 103 is linked to the annotation data for the pseudo SAR image. The linking of the pseudo SAR image to the annotation data for the pseudo SAR image may be performed by, for example, a file name. Furthermore, the pseudo SAR image may be linked to the annotation data for the pseudo SAR image by superimposing the annotation data on the pseudo SAR image. The specific linking method is not particularly limited.

[0032] The adding unit 104 can add the pseudo SAR image and the annotation data to the training data by, for example, storing the pseudo SAR image and the annotation data in a database in which the training data is stored. In the database, the pseudo SAR image and the annotation data may be stored separately. For example, the pseudo SAR image and the annotation data may be stored in separate folders.

[0033] Here, with reference to FIG. 2, a flow of adding training data for a pseudo SAR image will be described. FIG. 2 is a diagram showing an example of a flow of adding training data for a pseudo SAR image. The diagram on the top left of FIG. 2 is an image of an optical image acquired by the image acquisition unit 101. In the example shown in FIG. 2, the target object is a ship. The diagram in the top center of FIG. 2 is an image of a pseudo SAR image generated from the optical image by conversion processing. Moreover, the diagram in the bottom center of FIG. 2 is an image of annotation data related to the target object that is added to the optical image. That is, the diagram in the bottom center of FIG. 2 is an image of annotation data acquired by the annotation data acquisition unit 102. In the image of the annotation data, the part represented by a black rectangle is the position of the target object in the optical image. Moreover, in the image of the annotation data, the type of target object captured in the optical image is represented as a ship. Then, in FIG. 4, the pseudo SAR image and the annotation data are linked and added as training data. The adding unit 104 may add the pseudo SAR image and the annotation data to the training data separately. The pseudo SAR image on which the annotation data is superimposed may be added as the training data. The optical image, pseudo SAR image, and annotation data are not limited to the example shown in FIG. 4. Furthermore, the target object is not limited to a ship.

[0034] 3, the operation of the image generating device 10 including the image acquiring unit 101, the annotation data acquiring unit 102, the generating unit 103, and the adding unit 104 will be described.

[0035] In step S101, the image acquisition unit 101 acquires an optical image of an object. In step S102, the annotation data acquisition unit 102 acquires annotation data related to the object that has been added to the optical image. In step S103, the generation unit 103 generates a pseudo SAR image that simulates an SAR image from the optical image through conversion processing. In step S104, the addition unit 104 associates the pseudo SAR image with the annotation data and adds it to the training data. Then, the image generation device 10 ends its operation.

[0036] The image generating device 10 can repeat the above flow. For example, the image generating device 10 may generate pseudo SAR images as training data for a plurality of optical images. After step S104, the process returns to step S101, and the image acquiring unit 101 can acquire a new optical image. Then, the image generating device 10 performs the operations from step S102 to step S104 for the new optical image.

[0037] In step S101, if the image acquisition unit 101 acquires a plurality of optical images, the image generation device 10 may repeat the operations of steps S102 to S104 for each optical image.

[0038] In this embodiment, the image generating device 10 has an annotation data acquiring unit 102 that acquires annotation data related to an object that is assigned to an optical image. Then, a generating unit 103 generates a pseudo SAR image that simulates an SAR image from the optical image, and an adding unit 104 associates the pseudo SAR image with the annotation data and adds it to the training data. When annotating SAR images, it may be difficult to perform the annotation accurately. As a result, annotation errors may occur in SAR images. However, optical images may be more accurately annotated than SAR images. Therefore, by annotating optical images and adding pseudo SAR images generated from the optical images as training data, it is possible to reduce annotation errors when creating training data for SAR images.

[0039] In SAR images, for example, objects may be confused with noise. As a result, noise may be annotated as an object, resulting in an annotation error. Another example of an annotation error is when an object included in a SAR image is not annotated as an object. In other words, the object in the SAR image may be overlooked. However, optical images may be easier to accurately annotate than SAR images. Therefore, annotating optical images can reduce annotation errors. Training data is used, for example, to generate a model for detecting objects included in SAR images. Accurately annotated annotation data is saved as training data along with pseudo-SAR images, enabling improved learning accuracy using the training data.

[0040] Furthermore, the annotation data acquisition unit 102 may acquire annotation data that uses information that assists annotation, for example. For example, when the target object is a ship, the annotation data acquisition unit 102 may acquire annotation data that uses AIS information. Using information that assists annotation may enable accurate annotation. In other words, it is possible to reduce annotation errors when creating training data for SAR images.

[0041] Here, an example of the conversion process is a process of generating a pseudo SAR image from an optical image using a generative adversarial network. The generation unit 103 may be able to generate a highly accurate pseudo SAR image by generating a pseudo SAR image from an optical image using a generative adversarial network. Then, the addition unit 104 adds the highly accurate pseudo SAR image as training data, thereby enabling improvement in the accuracy of learning using the training data.

[0042] Furthermore, in the image generating device 10, adding pseudo SAR images generated from optical images as training data makes it possible to increase the amount of training data. In generating a model for detecting objects contained in SAR images, the amount of training data may be insufficient. Therefore, the generating unit 103 generates pseudo SAR images from optical images, and the adding unit 104 stores the pseudo SAR images as training data, making it possible to increase the amount of training data. Increasing the amount of training data makes it possible to improve the accuracy of learning using the training data.

[0043] [Variations] The modified example will be described in detail with reference to the drawings. Below, the description of the present embodiment will be omitted to the extent that it does not make the description unclear.

[0044] In this modification, the image generating device 10 adds SAR images as training data in addition to pseudo SAR images.

[0045] The image acquisition unit 101 acquires an SAR image of an object. The method of acquiring the SAR image by the image acquisition unit 101 is the same as the method of acquiring the optical image by the image acquisition unit 101.

[0046] The annotation data acquisition unit 102 acquires annotation data related to the object that has been assigned to the SAR image. The method for acquiring annotation data for the SAR image is the same as the method for acquiring annotation data for the optical image.

[0047] The adding unit 104 links the SAR image with the annotation data and adds it to the training data. As a result, the training data includes a pseudo SAR image and an SAR image. The method of linking the SAR image with the annotation data for the SAR image is the same as that for the pseudo SAR image. Furthermore, the method of adding the SAR image to the training data is the same as that for the pseudo SAR image. The ratio of SAR images to pseudo SAR images included in the training data is arbitrary. The ratio of SAR images to pseudo SAR images may be, for example, a ratio according to a model learned using the training data. Furthermore, the ratio of SAR images to pseudo SAR images may be, for example, a ratio according to an object captured in the image.

[0048] A flow of adding training data for SAR images will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of a flow of adding training data for SAR images. The diagram on the upper left of FIG. 4 is an image of an SAR image acquired by the image acquisition unit 101. In the example shown in FIG. 4, the object is a ship. The diagram on the lower left of FIG. 4 is an image of annotation data related to the object added to the SAR image. That is, the diagram on the lower left of FIG. 4 is an image of annotation data acquired by the annotation data acquisition unit 102. In the image of the annotation data, the part represented by a black rectangle is the position of the object in the SAR image. Furthermore, in the image of the annotation data, the type of object captured in the SAR image is represented as a ship. Then, in FIG. 4, the SAR image and the annotation data are linked and added as training data. The SAR image and the annotation data may be added separately to the training data. The SAR image on which the annotation data is superimposed may be added as annotation data. The SAR image and annotation data are not limited to the example shown in FIG.

[0049] In this embodiment, the image generating device 10 includes an annotation data acquiring unit 102 that acquires annotation data related to an object that is assigned to a SAR image, and an adding unit 104 that associates the SAR image with the annotation data and adds it to training data. By adding the SAR image as training data by the adding unit 104, the training data includes both pseudo SAR images and SAR images. Including not only pseudo SAR images generated from optical images but also SAR images in the training data enables improved learning accuracy in generating a model for detecting objects included in SAR images. Furthermore, adding SAR images to the training data enables an increase in the amount of training data.

[0050] [Second embodiment] The second embodiment will be described in detail with reference to the drawings. Below, the description of the second embodiment will be omitted to the extent that it does not make the description of the present embodiment unclear.

[0051] The configuration of the image generating device 20 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the configuration of the image generating device 20. The image generating device 20 includes an image acquiring unit 201, an annotation data acquiring unit 202, a generating unit 203, an adding unit 204, and an output unit 205. The image generating device 20 may also include a receiving unit 206.

[0052] The image acquisition unit 201, the annotation data acquisition unit 202, the generation unit 203, and the addition unit 204 are the same as those in the above embodiment.

[0053] The output unit 205 is one form of output means that outputs a screen that displays an optical image and a pseudo SAR image generated from the optical image in association with each other.

[0054] The output unit 205 outputs a screen to be displayed on, for example, a display device (not shown). The display device displays the screen output by the output unit 205. The user can then check the screen displayed on the display device. The display device and the image generating device 20 are connected via a wired or wireless network.

[0055] The output unit 205 outputs a screen that displays the pseudo SAR image generated by the generation unit 203. The user can, for example, check the pseudo SAR image displayed on a display device. FIG. 6 is a diagram showing an example of a screen that displays a pseudo SAR image. Referring to FIG. 6, a pseudo SAR image generated from an optical image is displayed. When multiple pseudo SAR images have been generated, the user can press "next image" or "previous image" displayed on the screen. By pressing "next image" or "previous image" displayed on the screen, the output unit 205 can display a pseudo SAR image that is different from the pseudo SAR image that is being displayed.

[0056] The output unit 205 may output a screen displaying the range of the object of the annotation data in the pseudo SAR image. Furthermore, the output unit 205 may output a screen displaying an enlarged view of the range of the object of the annotation data in the pseudo SAR image. The range of the object of the annotation data is, for example, the range represented by a rectangle in the example of annotation data shown in FIG. 2. The range of the object of the annotation data is not limited to this, and may be, for example, any range that includes the entire annotated object.

[0057] The output unit 205 may output a screen displaying a pseudo SAR image corresponding to a range specified in the optical image. For example, the image acquisition unit 201 can accept designation of a range to be displayed by the output unit 205 when acquiring the optical image. Then, the output unit 205 can output an image displaying a range of the pseudo SAR image corresponding to the range specified in the optical image. The output unit 205 may output an image displaying an enlarged range of the pseudo SAR image corresponding to the range specified in the optical image. Here, the designation of the range to be displayed by the output unit 205 may be accepted by a configuration different from the image acquisition unit 201. The designation of the range to be displayed by the output unit 205 may be accepted by, for example, the accepting unit 206 described below.

[0058] The output unit 205 outputs a screen that displays an optical image and a pseudo SAR image generated from the optical image in association with each other. The output unit 205 can output a screen that displays an optical image and a pseudo SAR image generated from the optical image in a manner that allows them to be compared. The output unit 205 can output, for example, a screen that displays an optical image and a pseudo SAR image generated from the optical image side by side. FIG. 7 is a diagram showing an example of a screen that displays an optical image and a pseudo SAR image generated from the optical image in association with each other. Referring to FIG. 7, the optical image and the pseudo SAR image generated from the optical image are displayed side by side. By displaying the optical image and the pseudo SAR image generated from the optical image side by side, the user can compare the optical image and the pseudo SAR image. The "Reconvert" button and "Add to Training Data" button included in the example screen of FIG. 7 will be described later.

[0059] The output unit 205 can also output a screen that switchably displays an optical image and a pseudo SAR image generated from the optical image. For example, a user performs a switching operation between an optical image and a pseudo SAR image generated from the optical image. The output unit 205 can switch the image displayed on the screen through a switching operation by the user. By switching the image, the user can compare the optical image and the pseudo SAR image generated from the optical image. For example, the output unit 205 may output a screen that switches between an optical image and a pseudo SAR image generated from the optical image every predetermined time. The predetermined time is a time period during which the user checks the image. The predetermined time period is set, for example, by the user.

[0060] The output unit 205 may output a screen displaying the range of the object of the annotation data in each of the optical image and the pseudo SAR image displayed in association with each other. The output unit 205 may output a screen displaying an enlarged view of the range of the object of the annotation data in each of the optical image and the pseudo SAR image displayed in association with each other. By enlarging and displaying the range of the object of the annotation data in each of the optical image and the pseudo SAR image, the user can compare the optical image and the pseudo SAR image with respect to the range of the object. The range of the object of the annotation data is as described above. The output unit 205 may output a screen displaying the range of the object of the annotation data in the optical image or the pseudo SAR image.

[0061] The output unit 205 may output a screen displaying a range corresponding to a range specified in the optical image for the pseudo SAR image displayed in association with the optical image. The output unit 205 may output a screen displaying an enlarged range corresponding to a range specified in the optical image for the pseudo SAR image. The specification of the range displayed by the output unit 205 is as described above. The output unit 205 may further output a screen displaying a range specified in the optical image. The output unit 205 may also output a screen displaying an enlarged range specified in the optical image.

[0062] When the generation unit 203 generates a pseudo optical image, the output unit 205 may output a screen displaying the optical image and a pseudo optical image generated based on the optical image. Here, the pseudo SAR image generated based on the optical image is a pseudo optical image generated by the generation unit 203 from the pseudo SAR image generated by the generation unit 203 from the optical image. The output unit 205 can output a screen displaying the optical image and the pseudo optical image in association with each other. The output unit 205 may, for example, output a screen displaying the optical image and the pseudo optical image in a comparable manner.

[0063] FIG. 8 is a diagram showing an example of a screen displaying an optical image and a pseudo optical image generated based on the optical image. The optical image and the pseudo optical image are displayed side by side on the screen. A user can compare the optical image and the pseudo optical image displayed on the screen. FIG. 8 is an example of a screen displaying an optical image and a pseudo optical image generated based on the optical image. The screen displaying the optical image and the pseudo optical image generated based on the optical image is not limited to this example. The output unit 205 may output a screen that switchably displays, for example, an optical image and a pseudo optical image generated based on the optical image.

[0064] The image generation device 20 may further include a reception unit 206 .

[0065] The reception unit 206 is one aspect of reception means that receives an instruction to add a pseudo SAR image to training data. The reception unit 206 receives an instruction to add the pseudo SAR image generated by the generation unit 203 to the training data. The reception unit 206 may receive the instruction to add the pseudo SAR image to the training data, for example, on a screen. The output unit 205 can output, for example, a screen that can receive an instruction to add the pseudo SAR image to the training data. Then, the user can issue an instruction to add the pseudo SAR image to the training data by performing an operation on the screen. That is, the reception unit 206 may receive the instruction to add the pseudo SAR image to the training data by a user's operation on the screen.

[0066] Here, reference is made again to FIG. 7. In the example screen in FIG. 7, an "Add to Training Data" button is displayed. When the user presses the "Add to Training Data" button, reception unit 206 can receive an instruction to add the pseudo SAR image to the training data. In other words, FIG. 7 is also an example of a screen capable of receiving an instruction to add the pseudo SAR image to the training data. The screen capable of receiving an instruction to add the pseudo SAR image to the training data is not limited to this. Output unit 205 may output a screen capable of receiving an instruction to add the pseudo SAR image to the training data, as a screen different from the screen displaying the pseudo SAR image.

[0067] The reception unit 206 may receive an instruction to add a pseudo SAR image to the training data by voice input. For example, a user may input an instruction to add a pseudo SAR image to the training data by voice to a voice input device (not shown). The voice input by the user to the voice input device is analyzed by voice recognition. The analysis by voice recognition is performed using known technology. Here, the voice input device and the image generation device 20 are connected by a wired or wireless network. The method of receiving an instruction to add a pseudo SAR image to the training data is not limited to these.

[0068] When the receiving unit 206 receives an instruction to add a pseudo SAR image as training data, the adding unit 204 links the pseudo SAR image with the annotation data and adds it to the training data. In a case where the image generating device 20 includes the receiving unit 206, when the receiving unit 206 receives an instruction not to add a pseudo SAR image as training data, the adding unit 204 may not add the pseudo SAR image to the training data. The receiving unit 206 can receive the instruction not to add the pseudo SAR image as training data, for example, on a screen. As a specific example, a "Do not add to training data" button may be displayed on the screen output by the output unit. Then, when the user presses the button, the instruction not to add the pseudo SAR image as training data can be received. The receiving unit 206 may receive the instruction not to add the pseudo SAR image as training data by voice input. Furthermore, if the receiving unit 206 does not receive an instruction to add a pseudo SAR image to the training data within a predetermined period of time, the receiving unit 206 may consider that it has received an instruction not to add a pseudo SAR image to the training data.

[0069] The receiving unit 206 may receive an instruction to generate a pseudo SAR image again from the optical image. When the user checks the pseudo SAR image generated by the generating unit 203, the user may want to generate the pseudo SAR image again by conversion processing. In other words, the user may want to generate the pseudo SAR image again. Therefore, the receiving unit 206 can receive an instruction to generate a pseudo SAR image again from the optical image.

[0070] The receiving unit 206 can receive, on a screen, an instruction to generate a pseudo SAR image again from an optical image. The output unit 205 can, for example, output a screen capable of receiving an instruction to generate a pseudo SAR image again from an optical image. Then, the user can issue an instruction to generate a pseudo SAR image again by performing an operation on the screen. In other words, the receiving unit 206 may receive an instruction to generate a pseudo SAR image again by a user's operation on the screen.

[0071] Here, reference is made again to FIG. 7. In the example screen in FIG. 7, a "reconvert" button is displayed. When the user presses the "reconvert" button, reception unit 206 can receive an instruction to generate a pseudo SAR image again. In other words, FIG. 7 is also an example of a screen that can receive an instruction to generate a pseudo SAR image again from an optical image. In the example screen shown in FIG. 7, an optical image and a pseudo SAR image are displayed. Therefore, if the user compares the optical image with the pseudo SAR image and considers that it is necessary to generate a pseudo SAR image again, the user can issue an instruction to generate a pseudo SAR image again.

[0072] The screen capable of accepting an instruction to regenerate a pseudo SAR image from an optical image is not limited to this. Output unit 205 may output a screen capable of accepting an instruction to regenerate a pseudo SAR image as a screen different from the screen displaying the pseudo SAR image.

[0073] The receiving unit 206 may receive an instruction to generate a pseudo SAR image again by voice input. The method of receiving an instruction to add a pseudo SAR image to training data is not limited to this.

[0074] When receiving unit 206 receives an instruction to generate a pseudo SAR image again, generating unit 203 generates a pseudo SAR image again from the optical image by conversion processing. Then, adding unit 204 may add the pseudo SAR image generated again to the training data. Furthermore, output unit 205 may output a screen that displays the optical image and the pseudo SAR image generated again from the optical image in association with each other. When receiving unit 206 receives an instruction not to generate a pseudo SAR image again, adding unit 204 may add the pseudo SAR image to the training data. Receiving unit 206 can receive the instruction not to generate a pseudo SAR image again, for example, on a screen or by voice input. Furthermore, when receiving unit 206 does not receive an instruction to generate a pseudo SAR image again, it may be considered that it has received an instruction not to generate a pseudo SAR image again. Furthermore, when output unit 205 outputs a screen capable of accepting an instruction to add a pseudo SAR image to training data, the acceptance of the instruction to add a pseudo SAR image to training data may be the acceptance of an instruction not to generate a pseudo SAR image again.

[0075] The receiving unit 206 may receive a designation of a range to be enlarged and displayed in the pseudo SAR image. First, the output unit 205 can output a screen on which the range designated in the pseudo SAR image can be enlarged and displayed. Then, for example, the user can designate a range to be enlarged on the screen on which the pseudo SAR image is displayed. At this time, the receiving unit 206 can receive a designation of a range to be enlarged and displayed in the pseudo SAR image on the screen. Then, the output unit 205 may output a screen on which the designated range in the pseudo SAR image is enlarged and displayed.

[0076] 9, the operation of the image generating device 20, which includes the image acquiring unit 201, the annotation data acquiring unit 202, the generating unit 203, the adding unit 204, and the output unit 205, will be described. FIG. 9 is a flowchart showing the operation of the image generating device 20.

[0077] Steps S201, S202, and S203 in Fig. 9 are the same as steps S101, S102, and S103 in Fig. 3, respectively. In step S204, output unit 205 outputs a screen displaying the optical image and the pseudo SAR image in association with each other. In step S205, adding unit 204 links the pseudo SAR image with the annotation data and adds it to the training data. Then, image generating device 20 ends its operation. Here, steps S204 and S205 can be performed in any order. That is, adding unit 204 may add the pseudo SAR image as training data, and then output unit 205 may output a screen displaying the optical image and the pseudo SAR image in association with each other. Furthermore, steps S204 and S205 may be performed in parallel.

[0078] The operation of the image generating device 20 will be described when the image generating device includes the receiving unit 206. After step S204, the receiving unit 206 receives an instruction to add the pseudo SAR image to the training data. Then, in step S205, the adding unit 204 links the pseudo SAR image with the annotation data and adds it to the training data. Then, the image generating device 20 ends its operation. If the receiving unit 206 receives an instruction not to add the pseudo SAR image to the training data, the image generating device 20 may end its operation.

[0079] After step S204, receiving unit 206 may receive an instruction to generate a pseudo SAR image again. If receiving unit 206 receives an instruction to generate a pseudo SAR image again, in step S203, generating unit 203 generates a pseudo SAR image again. If receiving unit 206 receives an instruction not to generate a pseudo SAR image again, in step S205, adding unit 204 adds the pseudo SAR image to the training data. Then, image generating device 20 ends its operation.

[0080] The image generating device 20 may repeat the above operations. For example, for one optical image, after adding a pseudo SAR image as training data in step S205, the process may return to step S201, and the image acquiring unit 201 may acquire a new optical image. Then, the image generating device 20 may perform the operations from step S202 to step S205 for the new optical image. Furthermore, if the image acquiring unit 201 acquires multiple optical images in step S201, the operations from step S202 to step S205 may be repeated for each optical image. Alternatively, after the image generating device 20 generates pseudo SAR images for multiple optical images, the output unit 205 may output a screen that displays the optical image and the pseudo SAR image in association with each other. That is, steps S201 to S203 are repeated for each of the multiple images. Then, in step S204, the output unit 205 may output a screen that displays the optical image and the pseudo SAR image in association with each other.

[0081] In this embodiment, the output unit 205 of the image generating device 20 outputs a screen displaying an optical image and a pseudo SAR image generated from the optical image in association with each other. A user may wish to check the pseudo SAR image generated from the optical image. A user may also wish to compare the pseudo SAR image with the optical image from which the pseudo SAR image was generated. Therefore, the output unit 205 outputs a screen displaying an optical image and a pseudo SAR image generated from the optical image in association with each other, allowing the user to check the optical image and the pseudo SAR image.

[0082] The output unit 205 may output a screen in which the range of the object of the annotation data in the pseudo SAR image is enlarged and displayed. The pseudo SAR image, which is training data, may be used, for example, to generate a model for detecting objects contained in the SAR image. In model learning, the way the object contained in the image is captured may be important. Therefore, a user may want to check whether the conversion process of the range containing the object in the pseudo SAR image has been performed successfully. Therefore, by the output unit 205 outputting a screen in which the range of the object of the annotation data is enlarged and displayed, a user can check the pseudo SAR image for the range containing the object. Furthermore, by enlarging and displaying the range of the object of the annotation data, a user may be able to carefully observe the range containing the object.

[0083] The output unit 205 may output a screen displaying a pseudo SAR image corresponding to a range specified in the optical image. If there is a range in the generated pseudo SAR image that the user wants to confirm, the user can specify that range in the optical image. Then, the output unit 205 outputs a screen displaying the pseudo SAR range corresponding to the specified range. This allows the user to confirm the specified range.

[0084] In this embodiment, the generation unit 203 of the image generation device 10 generates a pseudo optical image that simulates an optical image from the pseudo SAR image through conversion processing. Then, the output unit 205 outputs a screen that displays the optical image and the pseudo optical image. A user may want to confirm whether the conversion processing has been performed appropriately. To this end, the generation unit 203 generates a pseudo optical image by performing conversion processing on the pseudo SAR image generated from the optical image. Then, the output unit 205 outputs a screen that displays the optical image acquired by the image acquisition unit 201 and the pseudo optical image generated by the generation unit 203. The user can check the optical image and the pseudo optical image via the screen. The user can use the optical image and the pseudo optical image as references to determine whether the conversion processing has been performed appropriately.

[0085] In the present embodiment, the receiving unit 206 of the image generating device 20 receives an instruction to add a pseudo SAR image to training data. Then, when the adding unit 204 receives the instruction, the adding unit 204 links the pseudo SAR image with the annotation data and adds it to the training data. A user may wish to add a portion of the generated pseudo SAR image to the training data. For example, the user may want to check the generated pseudo SAR image and determine whether to add it to the training data. As a specific example, a user may wish to add only pseudo SAR images that the user himself / herself has determined have been appropriately converted to the training data. Therefore, when the receiving unit 206 receives an instruction to add a pseudo SAR image to the training data, the adding unit 204 adds the pseudo SAR image to the training data, thereby enabling addition to the training data according to the user's judgment. Furthermore, for example, by adding pseudo SAR images that the user has determined have been appropriately converted to the training data, the accuracy of the pseudo SAR images included in the training data may be improved.

[0086] In this embodiment, the receiving unit 206 of the image generating device 20 receives an instruction to generate a pseudo SAR image again from the optical image. Then, when the generating unit 203 receives the instruction, it generates a pseudo SAR image again from the optical image by conversion processing. A user may want to add to the training data only pseudo SAR images that the user himself / herself has determined have been converted appropriately. Then, when the user determines that the conversion processing has not been performed appropriately, the user may want the image generating device 20 to generate a pseudo SAR image again. In other words, the user may want the pseudo SAR image to be generated again. Therefore, when the receiving unit 206 receives an instruction to generate a pseudo SAR image again, the generating unit 203 generates a pseudo SAR image again from the optical image, thereby making it possible to generate a pseudo SAR image in line with the user's wishes. Furthermore, by generating a pseudo SAR image again, the accuracy of the pseudo SAR image may be improved.

[0087] [Hardware configuration example] 10 is a diagram illustrating an example of the hardware configuration of the image generation device 30 according to the present disclosure. The image generation device 30 is realized by a computer. The image generation device 30 is an example in which the image generation device 10 or the image generation device 20 is realized by a computer.

[0088] The image generating device 30 includes a processor 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a storage device 304 such as a hard disk for storing programs, an input / output interface 305 for inputting and outputting data, and a communication interface 306 for network connection. Each component is connected via a bus 307.

[0089] The processor 301 runs an operating system to control the entire computer. Examples of the processor 301 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The processor 301 loads programs stored in, for example, the ROM 302 or the storage device 304. The processor 301 then executes each process coded in the program. The processor 301 may execute processes or instructions in the illustrated flowchart based on the program.

[0090] The ROM 302 stores application programs, programs according to the embodiments, etc. The RAM 303 is used as a work area for the processor 301.

[0091] Examples of the storage device 304 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), etc. The storage device 304 stores, for example, an operating system (OS) program, application programs, and programs according to each embodiment.

[0092] The input / output interface 305 is connected to peripheral devices (not shown) via a wired network or a wireless network.

[0093] The communication interface 306 is connected to a communication network (not shown), such as a LAN (Local Network) or WAN (Wide Area Network), via a wireless or wired network. The communication network may be configured with multiple communication networks. This allows the computer to be connected to external devices via the communication network. The image generation device 30 may have components other than those shown in FIG. 10. For example, the image generation device 30 may have a drive device or the like. For example, the processor 301 may be attached to a drive device or the like and read programs and data stored in a non-transitory tangible recording medium into the RAM 303.

[0094] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with each other as long as they do not deviate from the scope of the present disclosure.

[0095] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0096] (Appendix 1) an image acquisition means for acquiring an optical image of an object; An annotation data acquisition means for acquiring annotation data relating to the object that is added to the optical image; a generating means for generating a pseudo SAR image that simulates a SAR (Synthetic Aperture Radar) image from the optical image by a conversion process; adding means for linking the pseudo SAR image with the annotation data and adding the pseudo SAR image to training data; An image generating device comprising:

[0097] (Appendix 2) The optical image and the pseudo SAR image generated from the optical image are displayed in association with each other on a screen. 2. The image generating device of claim 1.

[0098] (Appendix 3) The output means outputs a screen in which the range of the object of the annotation data in the pseudo SAR image is enlarged and displayed. 3. The image generating apparatus of claim 2.

[0099] (Appendix 4) The output means outputs a screen displaying the pseudo SAR image corresponding to a specified range in the optical image. 4. The image generating device according to claim 2 or 3.

[0100] (Appendix 5) The generating means generates a pseudo optical image that simulates the optical image from the pseudo SAR image by a conversion process. 5. The image generating apparatus of claim 4.

[0101] (Appendix 6) The output means outputs a screen that displays the optical image and the pseudo-optical image. 6. The image generating apparatus of claim 5.

[0102] (Appendix 7) further comprising a receiving means for receiving an instruction to add the pseudo SAR image to training data; When the adding means receives the instruction, the adding means links the pseudo SAR image with the annotation data and adds the pseudo SAR image to training data. 7. An image generating device according to any one of claims 2 to 6.

[0103] (Appendix 8) the receiving means receives an instruction to regenerate the pseudo SAR image from the optical image, When the generation means receives the instruction, the generation means regenerates the pseudo SAR image from the optical image by the conversion process. 8. The image generation device of claim 7.

[0104] (Appendix 9) the image acquisition means acquires an SAR image of the object; the annotation data acquisition means acquires annotation data about the object that is assigned to the SAR image; The adding means links the SAR image with the annotation data and adds it to the training data. 9. An image generating apparatus according to any one of claims 1 to 8.

[0105] (Appendix 10) The conversion process is a process of generating the pseudo-SAR image from the optical image using a generative adversarial network. 10. The image generating apparatus of any one of claims 1 to 9.

[0106] (Appendix 11) The annotation data acquisition means acquires the annotation data using information from an Automatic Identification System (AIS) when the object is a ship. 11. An image generating apparatus according to any one of claims 1 to 10.

[0107] (Appendix 12) Acquire an optical image of the object; acquiring annotation data relating to the object that is assigned to the optical image; generating a pseudo SAR image simulating a SAR (Synthetic Aperture Radar) image from the optical image through conversion processing; The pseudo SAR image is linked to the annotation data and added to the training data. Image generation method.

[0108] (Appendix 13) Acquire an optical image of the object; acquiring annotation data relating to the object that is assigned to the optical image; generating a pseudo SAR image simulating a SAR (Synthetic Aperture Radar) image from the optical image through conversion processing; The pseudo SAR image is linked to the annotation data and added to the training data. A program that causes a computer to perform a process.

[0109] (Appendix 14) Acquire an optical image of the object; acquiring annotation data relating to the object that is assigned to the optical image; generating a pseudo SAR image simulating a SAR (Synthetic Aperture Radar) image from the optical image through conversion processing; The pseudo SAR image is linked to the annotation data and added to the training data. A recording medium that stores a program that causes a computer to execute a process.

[0110] Some or all of the configurations described in Supplementary Notes 2-11, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 12-14 in the same manner as Supplementary Notes 2-11. Not limited to Supplementary Notes 1 and 12-14, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording devices for recording software, or systems, within the scope of each of the above-mentioned embodiments. [Explanation of symbols]

[0111] 10, 20, 30 Image generation device 101, 201 Image acquisition unit 102, 202 Annotation data acquisition unit 103, 203 generation section 104, 204 Additional section 205 Output section 206 Reception 301 processor 302 ROM 303 RAM 304 Storage device 305 Input / Output Interface 306 Communication Interface 307 Bus

Claims

1. an image acquisition means for acquiring an optical image of an object; An annotation data acquisition means for acquiring annotation data relating to the object that is added to the optical image; a generating means for generating a pseudo SAR image simulating a SAR (Synthetic Aperture Radar) image from the optical image by a conversion process; adding means for linking the pseudo SAR image with the annotation data and adding the pseudo SAR image to training data; An image generating device comprising:

2. The optical image and the pseudo SAR image generated from the optical image are displayed in association with each other on a screen. The image generating device of claim 1 .

3. The output means outputs a screen in which the range of the object of the annotation data in the pseudo SAR image is enlarged and displayed. The image generating device of claim 2 .

4. The output means outputs a screen displaying the pseudo SAR image corresponding to a specified range in the optical image. The image generating device of claim 2 .

5. The generating means generates a pseudo optical image that simulates the optical image from the pseudo SAR image by a conversion process. The image generating device of claim 1 .

6. a receiving unit for receiving an instruction to add the pseudo SAR image to training data; When the adding means receives the instruction, the adding means links the pseudo SAR image with the annotation data and adds the pseudo SAR image to training data.

6. An image generating device according to claim 1.

7. the receiving means receives an instruction to regenerate the pseudo SAR image from the optical image; When the generating means receives the instruction, the generating means generates the pseudo SAR image again from the optical image by the conversion process.

7. The image generating device of claim 6.

8. the image acquisition means acquires an SAR image of the object; the annotation data acquisition means acquires annotation data about the object that is assigned to the SAR image; The adding means links the SAR image with the annotation data and adds it to the training data. The image generating device of claim 1 .

9. Acquire an optical image of the object; acquiring annotation data relating to the object that is assigned to the optical image; generating a pseudo SAR image simulating a SAR (Synthetic Aperture Radar) image from the optical image by a conversion process; The pseudo SAR image is linked to the annotation data and added to training data. Image generation method.

10. Acquire an optical image of the object; acquiring annotation data relating to the object that is assigned to the optical image; generating a pseudo SAR image simulating a SAR (Synthetic Aperture Radar) image from the optical image by a conversion process; The pseudo SAR image is linked to the annotation data and added to training data. A program that causes a computer to perform a process.

Images