Image processing apparatus, image processing method, and image processing program

Abstract

This image processing device is configured to comprise a region setting unit, a standard image extraction unit, a data generation unit, and an output unit. The region setting unit sets, in an annotation target image, a region in which an annotation target object can be present as a candidate region. The standard image extraction unit extracts, from an image on which annotation has been completed, a standard image that is an image in which an object identical to the target object is captured. The data generation unit generates, as annotation data, data in which the annotation target image, a reference image which is captured of a region including the candidate region at a time different from the time when the annotation target image is captured, and the standard image are associated with each other. The output unit outputs the annotation data generated by the data generation unit.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and the like.

Background Art

[0002] For effective utilization of satellite images and the like, various automatic analyses are being performed. For automatic analysis of images, development of analysis methods, and performance evaluation, image data with correct answers prepared is required. Assigning correct answers to data is also called annotation. In order to improve the accuracy of automatic analysis, it is desirable to have a large amount of image data with correct answers. However, satellite images, particularly image data generated by synthetic aperture radar, are often difficult to discriminate in terms of content. Therefore, preparing image data with correct answers requires complicated and a lot of work. Against this background, it is desirable to have a system that streamlines the work of assigning correct answers to image data.

[0003] The change reading system of Patent Document 1 is a system that determines the presence or absence of an object loss by image processing. The change reading system of Patent Document 1 generates correct answer data indicating that a house in the image has disappeared based on the comparison result of two image data taken at different times.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1, when the object to be annotated is an object that is difficult to discriminate, the accuracy of assigning correct answers may decrease.

[0006] In order to solve the above problems, an object of the present invention is to provide an image processing apparatus and the like that can improve accuracy while efficiently performing annotation.

Means for Solving the Problems

[0007] In order to solve the above problems, an image processing apparatus according to the present invention includes: a region setting means for setting, as a candidate region, a region in a target image for annotation where a target object for annotation may exist; a reference image extraction means for extracting a reference image, which is an image in which an object identical to the target object is photographed, from an image in which annotation has been completed; a data generation means for generating, as annotation data, data associating the target image for annotation, a reference image photographed at a time different from the target image for annotation for a region including the candidate region, and the reference image; and an output means for outputting the annotation data generated by the data generation means.

[0008] An image processing method according to the present invention includes: setting, as a candidate region, a region in a target image for annotation where a target object for annotation may exist; extracting a reference image, which is an image in which an object identical to the target object is photographed, from an image in which annotation has been completed; generating, as annotation data, data associating the target image for annotation, a reference image photographed at a time different from the target image for annotation for a region including the candidate region, and the reference image; and outputting the generated annotation data.

[0009] The image processing program recorded on the recording medium of the present invention causes a computer to execute a process of setting, as a candidate region, a region where there may be an object to be annotated in the image to be annotated, a process of extracting a reference image, which is an image in which the same object as the object to be annotated is photographed, from an image for which annotation has been completed, a process of generating, as annotation data, data associating the image to be annotated, a reference image photographed at a time different from the image to be annotated for a region including the candidate region, and the reference image, and a process of outputting the generated annotation data.

Advantages of the Invention

[0010] According to the present invention, it is possible to improve the accuracy while efficiently performing annotation.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Embodiments for Carrying Out the Invention

[0012] (First Embodiment) The first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an outline of the configuration of the image processing system of the present embodiment. The image processing system includes an image processing apparatus 10 and a terminal apparatus 30. The image processing apparatus 10 and the terminal apparatus 30 are connected via a network.

[0013] The image processing system of this embodiment is a system that performs processing related to annotations on images. Processing related to annotations means, for example, outputting an image to be annotated, and based on the input results of the operator's operations, associating information for identifying an object existing in the image and information on the region where the object exists with the image. The information to be associated with the image may be either information for identifying an object existing in the image or information on the region where the object exists. Also, in the processing related to annotations, the information to be associated with the image is not limited to these. The image processing system performs processing related to annotations on, for example, an image acquired using Synthetic Aperture Radar (SAR). The data generated using the image processing system can be used, for example, as teacher data in machine learning.

[0014] The configuration of the image processing apparatus 10 will be described. FIG. 2 is a diagram showing an example of the configuration of the image processing apparatus 10. The image processing apparatus 10 includes a region setting unit 11, a region extraction unit 12, a reference image extraction unit 13, a data generation unit 14, an output unit 15, an input unit 16, and a storage unit 20.

[0015] The storage unit 20 includes a target image storage unit 21, a reference image storage unit 22, a region information storage unit 23, and an annotation result storage unit 24.

[0016] The region setting unit 11 sets, as a candidate region, a region where a target object for annotation may exist in the image to be annotated. In the following description, the image to be annotated, that is, the image to be processed for annotations, is also referred to as the target image.

[0017] The region setting unit 11 sets, in the target image, a region where there may be a target object as a candidate region. For example, the region setting unit 11 reads out a target image to be processed from the target image storage unit 21. The region setting unit 11 stores the range of the candidate region on the target image in the region information storage unit 23. The region setting unit 11 stores the range of the candidate region on the target image in the region information storage unit 23, represented by coordinates in the target image, for example. Also, information such as the photographed location and date and time is added to the target image.

[0018] FIG. 3 is a diagram showing an example of a target image. FIG. 3 is an example of image data photographed by a synthetic aperture radar. The elliptical and rectangular regions in FIG. 3 indicate regions where the reflected wave is different from the surroundings, that is, regions where there may be an object. In FIG. 3, the surroundings of the elliptical and rectangular regions correspond to the sea, for example. The gray region on the right side of FIG. 3 corresponds to land, for example.

[0019] The region setting unit 11 sets, for example, a region where the state of the reflected wave is different from the surroundings as a candidate region where there may be a target object. The region setting unit 11 identifies, for example, a region where the luminance is different from the surroundings in the target image, and sets a rectangular region including the identified region as a candidate region. FIG. 4 shows an example in which a candidate region is set as candidate region W on the target image. In the example of FIG. 4, the region setting unit 11 sets a rectangular region where the luminance is different from the surroundings as a candidate region where there may be a target object. In the example of FIG. 4, the candidate region W is set in the region surrounded by a dotted line from the upper right corner of the target image.

[0020] The region setting unit 11 identifies all locations in one target image where there may be a target object and sets them as candidate regions. The region setting unit 11 sets a plurality of candidate regions by sliding the candidate region in the target image, for example. The region setting unit 11 sets a plurality of candidate regions so as to cover the entire region of the candidate region existing in the target image, for example.

[0021] FIG. 5 and FIG. 6 are diagrams showing examples of operations for setting a plurality of candidate regions. For example, as shown in FIG. 5, the region setting unit 11 sequentially slides the candidate region W set in the upper left corner region of the target image to the right, and sets a plurality of candidate regions W. Further, as shown in FIG. 6, the region setting unit 11 may slide the candidate region W downward from the initial position in FIG. 5 and then sequentially slide it to the right to set a further plurality of candidate regions W. At this time, the candidate regions may or may not overlap each other. Further, the method of sliding the candidate region when setting a plurality of candidate regions is not limited to the above example. For example, when the candidate region is slid, the region setting unit 11 stores information indicating the range of the candidate region in the region information storage unit 23 if there is a region in the candidate region where the luminance change satisfies the criterion.

[0022] The region setting unit 11 may compare the position where the target image is acquired with the map information and set a candidate region within a region set in advance for the target image. For example, when the target object is a ship, it may be determined that the candidate region is set within a region where a ship may exist, such as the sea, a river, and a lake. In such a case, the region setting unit 11 refers to the map information, for example, and sets the candidate region only within the regions of the sea, the river, and the lake.

[0023] The region extraction unit 12 extracts an image of the region corresponding to the candidate region as a corresponding image from the reference image. Further, the region extraction unit 12 extracts an image of the region corresponding to the candidate region as a candidate image from the target image. The reference image is an image used as a comparison target for determining whether a target object exists in the target image. Further, the reference image is an image acquired at a time different from the target image in a region including the region of the target image. There may be a plurality of reference images corresponding to one target image.

[0024] The reference image is, for example, an image captured at a time different from the capture of the target image for an area including the area where the target image was captured, in the same manner as the target image. For example, for the same location, among the images captured at the same time every day, one image is set as the target image, and the images captured on other days are used as reference images. The cycle of image capture and the time of capture do not have to be constant. Information such as the location and date / time of capture is added to the reference image. The region extraction unit 12 reads the reference image from the reference image storage unit 22, for example.

[0025] Based on the information of the candidate regions stored in the region information storage unit 23, the region extraction unit 12 identifies the regions corresponding to the candidate regions on the reference image. The region extraction unit 12 extracts, from the reference image, the image of the region corresponding to the candidate region as the corresponding image.

[0026] The region extraction unit 12 may use the target image including the candidate region as the candidate image without extracting the candidate image from the target image. Also, the region extraction unit 12 may use the reference image including the candidate region as the corresponding image corresponding to the candidate region by referring to the position information added to the image without extracting the target image from the reference image.

[0027] FIG. 7 is a diagram showing an example of a reference image. Since the reference image in FIG. 7 is an image acquired at a time different from the target image, it shows an example in which the number of elliptical objects is different from that in the target image shown in FIG. 3. Also, FIG. 8 is a diagram showing an example of the candidate region W corresponding to the candidate region on the target image. The example of FIG. 8 shows the case where the region extraction unit 12 identifies the candidate region W set near the elliptical object on the reference image and extracts the image within the candidate region as the corresponding image.

[0028] The region extraction unit 12 extracts, for example, images of regions corresponding to candidate regions from two reference images. The two reference images are images respectively taken at times different from the target image. The region extraction unit 12 extracts, for example, an image within the candidate region of the target image as a corresponding image G2 and a corresponding image G3 for the candidate image G1. The region extraction unit 12 extracts, for example, the corresponding image G2 from the reference image A acquired one day before the day when the target image was acquired by the synthetic aperture radar, and extracts the corresponding image G3 from the reference image B acquired two days before. The region extraction unit 12 associates the candidate image G1, the corresponding image G2, and the corresponding image G3. The number of corresponding images associated with one candidate image does not have to be two, and is set according to the number of reference images. Also, the number of reference images can be set as appropriate.

[0029] The reference image extraction unit 13 extracts a reference image, which is an image in which an object identical to the target object is photographed, from the images for which annotation has been completed. The reference image extraction unit 13 searches the annotation result storage unit 24 for the image data of the images for which annotation has been completed and stored as annotation completed data, and extracts an image in which an object identical to the target object is photographed as the reference image. The identical objects include similar objects. As the reference image, for example, an image determined to have a correct result in the verification of the annotation result is used among the images for which annotation has been completed. The verification of the annotation result is performed using, for example, an optical image when the target image was photographed by a synthetic aperture radar.

[0030] For an image for which annotation has been completed, an image with an incorrect determination result at the time of annotation may be associated. An image with an incorrect determination result at the time of annotation is, for example, an image in which, when an image obtained by another method is used to determine an image for which annotation has been performed, the type of object determined in the annotation is found to be incorrect. For example, when annotating an object on an image obtained using synthetic aperture radar, assume that an object existing in a candidate region is determined to be a ship by an annotation operator. In the verification of the annotation result performed later, for example, when an object existing in the candidate region is identified as a tank by the annotation operator or another operator, the determination result at the time of annotation is determined to be incorrect. The verification of the annotation result is performed, for example, by an operator identifying an object existing in a candidate region using an optical image taken at the same location as the target image.

[0031] When an incorrect image is associated with the annotation completion data, the reference image extraction unit 13 extracts the image for which annotation has been completed and the incorrect image associated with the image for which annotation has been completed as reference images. In such a case, the reference image extraction unit 13, for example, regards the image for which annotation has been completed as a correct image and extracts a pair of the correct image and the incorrect image as a reference image.

[0032] The reference image extraction unit 13 compares the similarity between the candidate image and the image for which annotation has been completed, and determines that they are similar when the similarity is equal to or higher than a reference.

[0033] The reference image extraction unit 13 determines whether the object of the candidate image and the object of the image for which annotation has been completed are the same, for example, based on the similarity of map coordinates and the similarity of image feature amounts. The reference image extraction unit 13 may determine whether the object of the candidate image and the object of the image for which annotation has been completed are the same based on items other than those described above.

[0034] The reference image extraction unit 13 determines, for example, whether the object being photographed is the same as that in the candidate image for an image with completed annotation for which the photographing position is determined to be the same. The reference image extraction unit 13 determines whether the candidate image and the photographing position of the image with completed annotation are the same, for example, based on the distance between the center coordinates of the candidate image and the image with completed annotation. The reference image extraction unit 13 determines that the photographing positions are the same when the distance between the center coordinates of the candidate image and the image with completed annotation is equal to or less than a reference.

[0035] When the reference image extraction unit 13 determines whether the object in the candidate image and the object in the image with completed annotation are the same based on the similarity of image feature amounts, for example, it calculates the similarity of image feature amounts between images using feature point matching. In feature point matching, the reference image extraction unit 13 extracts feature points from, for example, the candidate image and the image with completed annotation, and determines that the two images are images of the same object when the similarity of the feature points meets the reference. The method for determining the similarity of image feature amounts using feature point matching is disclosed, for example, in P.F. Alcantarilla, J.Nuevo and A.Bartoli, “Fast Explicit Diffusion for Accelerated Features in Nonlinear Scale Spaces”, Proceedings British Machine Vision Conference 2013, pp.13.1-13.11. The reference image extraction unit 13 may calculate the similarity of image feature amounts using a method other than the feature point matching method. The reference image extraction unit 13 may calculate the similarity of image feature amounts using, for example, luminance histogram comparison or template matching.

[0036] When an image taken at the same point as the candidate region is not stored in the annotation result storage unit 24, the reference image extraction unit 13 compares each of the images of the annotation completion data stored in the annotation result storage unit 24 with the image to be annotated, and extracts a reference image. For example, when there is no image with the same shooting position, the reference image extraction unit 13 extracts, as a reference image, an image that satisfies the criterion of similarity of image feature amounts among all the images for which all the annotations stored in the annotation result storage unit 24 have been completed.

[0037] The reference image extraction unit 13 may further determine whether the object in the candidate image is the same as the object in the image for which the annotation has been completed, using the similarity of the sizes of the objects existing in the images. In such a case, for each pair of the two images, for example, the relationship between the number of pixels and the actual distance is set in advance. The reference image extraction unit 13 determines, for example, the similarity of the sizes of the objects on the two images based on the ratio or difference of the areas of the objects existing in the respective images. When the criterion of size similarity is set based on, for example, the ratio of the areas of the objects existing on the two images, the reference image extraction unit 13 determines that the sizes of the objects on the two images are the same when the ratio of the areas is within the reference range. Also, when the criterion of size similarity is set based on, for example, the difference in the areas of the objects existing on the two images, the reference image extraction unit 13 determines that the sizes of the objects on the two images are the same when the difference in the areas is within the reference.

[0038] The data generation unit 14 generates data associating the target image for annotation, the reference image, and the reference image as annotation data. The reference image is an image taken when the region including the candidate region is different from the target image for annotation. The data generation unit 14 generates annotation data by associating, for example, the target image for annotation, that is, the candidate image obtained by extracting the candidate region from the target image, the corresponding image obtained by extracting the candidate image from the reference image, and the reference image. The data generation unit 14 may generate annotation data by associating the candidate image, the corresponding image, and an image obtained by enlarging the vicinity of the candidate image among the candidate image, the corresponding image, and the reference image. The data generation unit 14 outputs the generated annotation data to the terminal device 30 via, for example, the output unit 15.

[0039] The data generation unit 14 may generate display data for displaying the candidate image, the corresponding image, and the reference image so that they can be compared as annotation data. The display data for displaying so that they can be compared refers to, for example, display data in a state where the images to be compared are arranged side by side in the horizontal direction so that an operator can compare and compare the two images. The data generation unit 14 may output the generated display data to a display device (not shown) connected to the image processing apparatus 10.

[0040] When an incorrect image is associated with the reference image extracted by the reference image extraction unit 13, the data generation unit 14 may generate annotation data with the reference image as a set of a correct image and an incorrect image.

[0041] FIG. 9 shows an example of a display screen for displaying the candidate image, the corresponding image, and the reference image output as annotation data so that they can be compared. In the example of FIG. 9, the reference image is displayed as a set of a correct image and an incorrect image. The image of P1 in FIG. 9 is the correct image among the reference images. The image of N1 in FIG. 9 is the incorrect image among the reference images. In the example of FIG. 9, information on the size of the target object and the position where the image was taken is added to the images of P1 and N1. The items of information added to the reference image are not limited to these.

[0042] The image of G1 in FIG. 9 is a candidate image, that is, an image corresponding to a candidate region on the target image. The images of G2 and G3 in FIG. 9 are corresponding images, that is, images corresponding to the candidate region on the reference image. The images of G2 and G3 are images on the reference images taken at different times respectively.

[0043] The data generation unit 14 generates annotation completion data based on the information regarding annotation. The information regarding annotation is input into the terminal device 30 as annotation information by the operation of an operator. The annotation information is, for example, information specifying the type of the target object in the target image, that is, the annotation target image, and information specifying the region where the object exists in the image. The data generation unit 14 acquires, for example, the information specifying the region where the object exists among the annotation information as a rectangular region surrounding the object in the candidate image. The data generation unit 14 generates, for example, data associating the type of the object in the candidate image and the information of the region where the object exists with the candidate image as annotation completion data based on the annotation information. The region indicated by the annotation information is also referred to as an annotation region. The data generation unit 14 stores the generated annotation completion data in the annotation result storage unit 24. The setting of the annotation region is not limited to the method of surrounding the region with a rectangular line. For example, the annotation region may be set by filling the annotation region.

[0044] FIG. 10 shows an example of the display screen when the annotation region is set on the display screen of FIG. 9. In FIG. 10, in the candidate image G1, a rectangular line is set as the line indicating the annotation region around the elliptical object. Also, in FIG. 10, rectangular lines are displayed at the corresponding positions for G2 and G3 which are the corresponding images.

[0045] FIG. 11 is a diagram showing only the lower part of the display screen in FIG. 9. FIG. 11 is a diagram showing an example of a display screen for displaying candidate image G1, corresponding image G2, and corresponding image G3 in a comparable manner. FIG. 12 shows, in the image of FIG. 11, an area where an object exists in candidate image G1 but does not exist in corresponding image G2 and corresponding image G3, indicated by a dotted line. By displaying candidate image G1 and corresponding images G2 and G3, which are different from the time of shooting, in a comparable manner in this way, the operator can more clearly recognize the area where the movable object exists.

[0046] FIG. 13 shows an example in which an annotation area is set on candidate image G1 by the operator's operation while the screen of FIG. 11 is being displayed. In FIG. 13, the area surrounded by a rectangular line on candidate image G1 is set as the annotation area. FIG. 14 shows an example of a display screen in which the annotation area is further displayed not only on candidate image G1 but also on corresponding image G2 and corresponding image G3. By displaying the annotation area not only on candidate image G1 but also on corresponding images G2 and G3, which are different from the time of shooting, in this way, the operator can perform the annotation process while more clearly recognizing the area where the movable object exists.

[0047] Output unit 15 outputs the annotation data generated by data generation unit 14 to terminal device 30. Output unit 15 may output display data generated based on the annotation data to a display device (not shown) connected to image processing device 10.

[0048] Input unit 16 receives, as annotation information, input of information regarding annotation of a target object for the annotation target image. Input unit 16 acquires the annotation information input to terminal device 30 by the operator's operation from terminal device 30.

[0049] The input unit 16 acquires, for example, as annotation information, information on the range of the annotation area and information for specifying the type of object in the image. The input unit 16 may acquire either one of the information on the range of the annotation area and the information for specifying the type of object in the image as the annotation information. Further, the input unit 16 may acquire information on items other than the above as the annotation information. The input unit 16 may acquire the annotation information from an input device (not shown) connected to the image processing apparatus 10.

[0050] The target image storage unit 21 of the storage unit 20 stores the image data of the image to be annotated as the target image. The target image storage unit 21 stores, for example, by associating the shooting date and time and the information on the shooting position with the target image. The reference image storage unit 22 stores the image data of the reference image. The reference image storage unit 22 stores, for example, by associating the shooting date and time and the information on the shooting position with the reference image. The reference image storage unit 22 may store by associating the information on the target image corresponding to the reference image. Further, the information associated with the target image and the reference image is not limited to these examples. The area information storage unit 23 stores the information on the range of the candidate area set by the area setting unit 11. The annotation result storage unit 24 stores the image to be annotated and the annotation information in association with each other as the annotation completion data. The annotation result storage unit 24 may store by associating the information on the shooting position of the image with the image included in the annotation completion data. Further, the annotation result storage unit 24 may store by associating the incorrect image with the image included in the annotation completion data.

[0051] Each of the above data regarding the annotation stored by the storage unit 20 is input to the image processing apparatus 10 by, for example, an operator. Each of the data regarding the annotation stored by the storage unit 20 may be acquired from a terminal device 30 or a server connected via a network.

[0052] The storage unit 20 is configured using, for example, a hard disk drive. The storage unit 20 may be configured using other storage devices such as, for example, a non-volatile semiconductor memory device. Further, the storage unit 20 may be configured by combining a plurality of types of storage devices such as a non-volatile semiconductor memory device and a hard disk drive. Further, part or all of the storage unit 20 may be provided in an external device connected to the image processing apparatus 10 via a network.

[0053] The terminal device 30 is a terminal device for operator operation and includes an input device and a display device (not shown). The terminal device 30 acquires annotation data from the image processing apparatus 10. The terminal device 30 outputs, to a display device (not shown), a display screen for performing annotation work based on the annotation data. The terminal device 30 displays, for example, a display screen in which a candidate image, a corresponding image, and a reference image are associated on the display device. The terminal device 30 may display both a correct image and an incorrect image for the reference image.

[0054] The terminal device 30 receives annotation information input by an operator's operation. The terminal device 30 outputs the acquired annotation information to the image processing apparatus 10. Further, there may be a plurality of terminal devices 30. The number of terminal devices 30 can be set as appropriate.

[0055] The operation of the image processing system of the present embodiment will be described. FIG. 15 is a diagram showing an example of an operation flow of the image processing apparatus 10 of the present embodiment.

[0056] The region setting unit 11 of the image processing apparatus 10 reads out a target image, which is an image for annotation, from the target image storage unit 21 of the storage unit 20.

[0057] When the target image is read, the region setting unit 11 sets, as candidate regions, regions on the target image where there may be objects to be annotated (step S11). The region setting unit 11 identifies, for example, regions where there may be objects based on the luminance values of the respective pixels in the image. When a region where there may be an object is identified, a rectangular region including the identified region is set as a candidate region. The region setting unit 11 sets, for example, a region smaller than the entire target image as a candidate region.

[0058] When candidate regions are set, the region setting unit 11 stores the information of the set candidate regions in the region information storage unit 23. The region setting unit 11 stores, for example, the coordinates specifying the outer peripheral portion of the candidate region on the target image in the region information storage unit 23 as the information of the candidate region.

[0059] The region setting unit 11 sets a plurality of candidate regions so as to cover the entire area of the candidate regions existing in the target image. The region setting unit 11 slides the candidate regions within the target image, for example, and sets regions where there may be objects as candidate regions.

[0060] When candidate regions are set, the region extraction unit 12 selects, as candidate regions to be annotated, the candidate regions stored in the region information storage unit 23 (step S12). The region extraction unit 12 selects, for example, the candidate region that was stored earliest as a candidate region among the candidate regions for which annotation is incomplete as the candidate region to be annotated. The method of selecting candidate regions may be other methods.

[0061] When a candidate region is selected, the region extraction unit 12 extracts, as a candidate image, the image of the portion of the candidate region from the target image. Also, the region extraction unit 12 reads out the reference image corresponding to the target image from the reference image storage unit 22 and extracts, as a corresponding image, the image of the portion of the candidate region from the reference image (step S13). The region extraction unit 12 extracts, for example, the images of the portions within the candidate region from two reference images as corresponding images respectively.

[0062] When the candidate image and the corresponding image are extracted, the reference image extraction unit 13 searches the annotation completion data stored in the annotation result storage unit 24, and extracts, as a reference image, an image in which the same object as the candidate image exists on the image (step S14). For example, the reference image extraction unit 13 extracts, as a reference image, an image whose similarity meets the criterion based on the similarity between the image stored as the annotation completion data and the candidate image.

[0063] When the reference image is extracted, the data generation unit 14 generates, as annotation data, data associating the image to be annotated, a reference image taken when the region including the candidate region is different from the image to be annotated, and the reference image (step S15). For example, the data generation unit 14 generates, as annotation data, data associating the candidate image, the corresponding image, and the reference image.

[0064] When the annotation data is generated, the output unit 15 outputs the generated annotation data to the terminal device 30 (step S16).

[0065] When acquiring the annotation data, the terminal device 30 outputs display data based on the annotation data to a display device (not shown). When annotation information is input by the operation of an operator while the display data based on the annotation data is being displayed, the terminal device 30 outputs the input annotation information to the image processing device 10.

[0066] The input unit 16 of the image processing device 10 acquires the annotation information from the terminal device 30 (step S17). When the annotation information is acquired, the data generation unit 14 generates annotation completion data by associating the data of the candidate image with the annotation information (step S18). The data generation unit 14 stores the generated annotation completion data in the annotation result storage unit 24.

[0067] When the annotation completion data is saved and the processing of the annotation is completed for all candidate regions (Yes in step S19), the image processing apparatus 10 ends the processing related to the annotation. When there is a candidate region for which the annotation processing has not been completed (No in step S19), the image processing apparatus 10 executes the processing from the operation of selecting the candidate region in step S12.

[0068] The annotation completion data generated by the above method can be used, for example, as teacher data when generating a machine learning model for identifying an image in an image recognition apparatus.

[0069] The above description has been given for an example of performing annotation on a target image acquired by a synthetic aperture radar, but the target image may be an image acquired by a method other than a synthetic aperture radar. For example, the target image may be an image acquired by an infrared camera.

[0070] The above description has shown an example of performing annotation with reference to an image acquired by the same method as the target image. In addition to such a configuration, it is also possible to verify the determination result in the annotation with reference to other types of images. For example, an image with completed annotation acquired by a synthetic aperture radar and an optical image captured by an optical camera that captures the visible light region at the same location may be displayed side by side to verify whether the type of the object determined in the annotation is correct. Also, by verifying whether it is correct in such a method, it is possible to generate a correct image and an incorrect image to be used as a reference image.

[0071] FIG. 16 is a diagram showing an example of a display screen when verifying a determination result in annotation. The example of FIG. 16 shows an example of a display screen that arranges and displays an image G1 for which annotation has been completed and an image V1 acquired by an imaging device different from the image G1 at the same point as the image G1 so as to be comparable. In the example of FIG. 16, the image G1 is, for example, an image acquired using a synthetic aperture radar, and the image V1 is, for example, an image acquired using an optical camera. Also, in the example of FIG. 16, selection buttons for "Next Image", "Correct Answer", and "Incorrect Answer" are set. The selection button for "Next Image" is a button for switching the image to be verified. "Correct Answer" is a button for inputting that the determination result in the annotation was correct. "Incorrect Answer" is a button for inputting that the determination result in the annotation was incorrect.

[0072] In the example of FIG. 16, for example, when the "Incorrect Answer" button is selected, the data generation unit 14 removes the corresponding data from the annotation completion data and stores it as incorrect answer data. In the example of FIG. 16, for example, when the "Correct Answer" button is selected, the data generation unit 14 associates information indicating that it was the correct answer with the annotation completion data and updates the annotation completion data. The data generation unit 14 may, for example, associate the image of the annotation completion data for which an incorrect answer was selected as an incorrect answer image with other annotation completion data including an image at the same point.

[0073] The image processing apparatus 10 of the image processing system according to this embodiment outputs annotation data by associating a candidate image obtained by extracting a region where an object may exist from a target image that is an image to be annotated, a corresponding image obtained by extracting a region corresponding to the candidate image from a reference image, and a reference image. The image processing apparatus 10 outputs, as annotation data, a target image that is an image to be annotated, an image taken at a different time from the target image, and a reference image for which annotation of the same object as the target object has been completed, in association with each other. By displaying each image so that they can be compared using the annotation data, for example, an operator performing annotation can perform the annotation work with reference to the presence or absence of changes in the target object and past annotation results, and can easily distinguish between the object and the region. In addition, when performing annotation, by displaying the reference image and the reference image, it is possible to suppress variations in judgment among the same operator and among operators. As a result, by using the image processing apparatus 10 according to this embodiment, it is possible to improve the accuracy while efficiently performing annotation.

[0074] When the image processing apparatus 10 outputs an incorrect image in the past annotation result as the reference image, the operator can refer to an example of a mistake when performing annotation. Therefore, for example, it may be easier to determine the type of the target object. Therefore, when the image processing apparatus 10 outputs an incorrect image in the past annotation result as the reference image, the accuracy of annotation can be further improved.

[0075] (Second Embodiment) A second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 17 is a diagram showing an outline of the configuration of the image processing apparatus 100. The image processing apparatus 100 of the present embodiment includes a region setting unit 101, a reference image extraction unit 102, a data generation unit 103, and an output unit 104. The region setting unit 101 sets, as a candidate region, a region in the annotation target image where the annotation target object may exist. The reference image extraction unit 102 extracts a reference image, which is an image in which the same object as the target object is photographed, from the image for which annotation has been completed. The data generation unit 103 generates, as annotation data, data associating the annotation target image, a reference image photographed at a time different from the annotation target image for a region including the candidate region, and the reference image. The output unit 104 outputs the annotation data generated by the data generation unit 103.

[0076] The region setting unit 11 is an example of the region setting unit 101. Further, the region setting unit 101 is an aspect of the region setting means. The reference image extraction unit 13 is an example of the reference image extraction unit 102. Further, the reference image extraction unit 102 is an aspect of the reference image extraction means. The data generation unit 14 is an example of the data generation unit 103. Further, the data generation unit 103 is an aspect of the data generation means. The output unit 15 is an example of the output unit 104. Further, the output unit 104 is an aspect of the output means.

[0077] The operation of the image processing apparatus 100 will be described. FIG. 18 is a diagram showing an example of the operation flow of the image processing apparatus 100. The area setting unit 101 sets, as a candidate area, an area in the annotation target image where the annotation target object may exist (step S101). When the candidate area is set, the reference image extraction unit 102 extracts, from the images in which the same object as the target object has been photographed, a reference image that is an image with annotation already completed (step S102). When the reference image is extracted, the data generation unit 103 generates, as annotation data, data associating the annotation target image, a reference image taken at a time different from the annotation target image for the area including the candidate area, and the reference image (step S103). When the annotation data is generated, the output unit 104 outputs the annotation data generated by the data generation unit 103 (step S104).

[0078] The image processing apparatus 100 of the present embodiment outputs, as annotation data, data associating the annotation target image, a reference image taken at a time different from the annotation target image for the area including the candidate area, and a reference image that is an image with annotation already completed. Therefore, by using the image processing apparatus 100, an operator can perform processing while comparing each image when performing annotation. As a result, by using the image processing apparatus 100 of the present embodiment, it is possible to improve the accuracy while efficiently performing the annotation process.

[0079] Each process in the image processing apparatus 10 of the first embodiment and the image processing apparatus 100 of the second embodiment can be performed by executing a computer program on a computer. FIG. 19 shows an example of the configuration of a computer 200 that executes a computer program for performing each process in the image processing apparatus 10 of the first embodiment and the image processing apparatus 100 of the second embodiment. The computer 200 includes a CPU (Central Processing Unit) 201, a memory 202, a storage device 203, an input / output I / F (Interface) 204, and a communication I / F 205.

[0080] The CPU 201 reads out and executes a computer program for each process from the storage device 203. The CPU 201 may be configured by a combination of a CPU and a GPU (Graphics Processing Unit). The memory 202 is composed of a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores the computer program executed by the CPU 201 and the data being processed. The storage device 203 stores the computer program executed by the CPU 201. The storage device 203 is composed of, for example, a non-volatile semiconductor storage device. Other storage devices such as a hard disk drive may be used for the storage device 203. The input / output I / F 204 is an interface for receiving inputs from an operator and outputting display data and the like. The communication I / F 205 is an interface for transmitting and receiving data to and from each device constituting the monitoring system. Also, the terminal device 30 can have a similar configuration.

[0081] The computer program used for the execution of each process can also be stored in a recording medium for non-temporary recording and distributed. As the recording medium, for example, a magnetic tape for data recording or a magnetic disk such as a hard disk can be used. Also, as the recording medium, an optical disk such as a CD-ROM (Compact Disc Read Only Memory) can be used. A non-volatile semiconductor storage device may be used as the recording medium.

[0082] The present invention has been described above by taking the above-described embodiments as examples. However, the present invention is not limited to the above-described embodiments. That is, within the scope of the present invention, various aspects understandable by those skilled in the art can be applied.

[0083] This application claims the priority based on Japanese Patent Application No. 2021-158568 filed on September 29, 2021, and incorporates all of its disclosures herein.

Explanation of Reference Numerals

[0084] 10 Image processing apparatus 11 Region setting unit 12 Region extraction unit 13 Reference image extraction unit 14 Data generation unit 15 Output unit 16 Input unit 20 Memory unit 21 Target image memory unit 22 Reference image memory unit 23 Region information memory unit 24 Annotation result memory unit 30 Terminal device 100 Image processing apparatus 101 Region setting unit 102 Reference image extraction unit 103 Data generation unit 104 Output unit 200 Computer 201 CPU 202 Memory 203 Storage device 204 Input / output I / F 205 Communication I / F

Claims

1. Region setting means for setting, as a candidate region, a region in the target image of annotation where there may be a target object of annotation; Reference image extraction means for extracting, from an image in which annotation has been completed, a reference image that is an image in which the same object as the target object has been photographed; Data generation means for generating, as annotation data, data associating the target image of annotation, a reference image photographed at a time different from the target image of annotation for a region including the candidate region, and the reference image; Output means for outputting the annotation data including, as the reference image, an image with a correct determination result and an image with an incorrect determination result when annotation is performed; An image processing apparatus comprising the above.

2. Input means for receiving, as annotation information, input of information regarding annotation of the target object on the target image of annotation; Storage means for storing, in association with each other, the target image of annotation and the annotation information as annotation completion data; The image processing apparatus according to claim 1, further comprising the above.

3. The storage means further stores, in association with the image included in the annotation completion data, information on the shooting position of the image; The reference image extraction means compares an image taken at the same point as the candidate region among the images stored in the storage means with the image of the candidate region of the target image of annotation, and extracts the reference image. The image processing apparatus according to claim 2.

4. When an image taken at the same point as the candidate region is not stored in the storage means, the reference image extraction means compares each image of the annotation completion data stored in the storage means with the target image of annotation, and extracts the reference image. The image processing apparatus according to claim 3.

5. The target image is an image captured using synthetic aperture radar. The image processing apparatus according to any one of claims 1 to 4.

6. The apparatus further comprises region extraction means for extracting, as a corresponding image, an image of a region corresponding to the candidate region from the reference image. The output means outputs, as the annotation data, data associating the image of the candidate region, the corresponding image, and the reference image. The image processing apparatus according to claim 1.

7. The region setting means sets a plurality of the candidate regions by sliding the position of the candidate region on the target image of the annotation. The image processing apparatus according to claim 1.

8. The region setting means sets the candidate region at a position where the target object may exist based on map information and the type of the target object. The image processing apparatus according to claim 1.

9. A computer sets, in a target image for annotation, a region where a target object for annotation may exist as a candidate region, extracts, as a reference image, an image in which an object identical to the target object is photographed from an image for which annotation has been completed, generates, as annotation data, data associating the target image for annotation, a reference image photographed at a time different from that of the target image for a region including the candidate region, and the reference image, outputs, as the annotation data, the annotation data including, as the reference image, an image in which a determination result at the time of performing annotation is correct and an image in which the determination result is incorrect. An image processing method.

10. In the target image for annotation, a process of setting, as a candidate region, a region where the target object for annotation may exist, a process of extracting, from an image in which annotation has been completed, a reference image that is an image in which the same object as the target object is photographed, a process of generating, as annotation data, data associating the target image for annotation, a reference image taken at a time different from the target image for annotation for a region including the candidate region, and the reference image, a process of outputting the annotation data including, as the reference image, an image in which the determination result at the time of annotation is correct and an image in which the determination result is incorrect An image processing program for causing a computer to execute.

Images