Training device, angle estimation device, training method, and angle estimation method

Abstract

The training device includes an angle difference calculation unit that calculates a difference between angles estimated by one or more angle estimators, a rigid transformation unit that transforms features of an input image according to the difference, a matching loss calculation unit that calculates a matching loss between non-transformed features extracted by one or more feature extractors and features transformed by the rigid transformation unit, and an update unit that updates at least one or more angle estimators by referring to the matching loss, and the rigid transformation unit transforms the features so that they appear as if they were extracted from an image with the same angle as the image from which the non-transformed features were extracted.

Description

[Technical Field]

[0001] The present invention relates to a training device, an angle estimation device, a training method, and an angle estimation method. [Background technology]

[0002] Angular information is useful for training object classifiers in SAR (synthetic aperture radar), especially when data is limited. Angular information is also useful for tasks related to optical imagery, such as face and object recognition.

[0003] An angle estimator can estimate angle information from an input image. Angular information includes, but is not limited to, the pose of an object in the image and the shooting angle of the camera that generated the image. Angular information can improve the performance of a classifier. For example, a classifier trained on only two images, A and B, without knowing the angle information will only work for images at angles A and B. However, a classifier trained on the same images with the angle information can also work for images at angles other than A and B by interpolating the image information between angles A and B. In face and object recognition, angle information can be used to predict what an object will look like from a new angle, even without training data for that new angle.

[0004] To improve the accuracy of the angle estimator, the angle estimator is trained using machine learning. One method uses correct angle labels to train the angle estimator (see, for example, Non-Patent Document 1). In this method, an image and its correct image label are first input. Next, the angle estimator, which is a trainable neural network, estimates the angle from the image. Furthermore, the angle estimator is updated according to a penalty. After thousands of repeated updates, the angle estimator estimates an angle that matches the correct angle label.

[0005] There is also a method that uses ground truth structural data (see, for example, Non-Patent Document 2). In this method, an image and its ground truth structural data are first input. Next, features are extracted from the two-dimensional (2D) image using a trainable feature extractor. Furthermore, angles are estimated from the 2D image using a trainable angle extractor. Furthermore, the structural data is projected according to the estimated angles to obtain projected features. Then, a penalty is calculated as the difference between the extracted features and the projected features. After that, the feature extractor and angle estimator are updated according to the penalty. After thousands of repeated updates, the angle estimator estimates the angle so that the projected features match the extracted features. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] S Tulsiani et al., "Viewpoints and keypoints", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1510-1519 [Non-patent document 2] G Pavlakos et al., "6-dof object pose from semantic keypoints", May 2017, In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2011-2018 Summary of the Invention [Problem to be solved by the invention]

[0007] The method using ground truth angle labels requires guidance from angle labels. The method using ground truth structure data requires guidance from structure data. It is desirable to train an angle estimator without guidance from angle labels or structure data.

[0008] Therefore, an object of the present invention is to provide a training device, an angle estimation device, a training method, and an angle estimation method that are capable of learning a model related to angle estimation without using annotations. [Means for solving the problem]

[0009] Exemplary embodiments of the training device include: Multiple Extract features from the input image multiple Feature extraction method and input image Camera shooting to generate Estimate the angle multiple An angle estimation means; multiple and an angle difference calculation means for calculating a difference between the angles estimated by the angle estimation means. Extracted from a rigid transformation means for transforming the features; Either extracted by the feature extraction means Features and a matching loss calculation means for calculating a matching loss between the feature transformed by the rigid body transformation means and the feature transformed by the rigid body transformation means; and multiple and an update means for updating the angle estimation means, The features extracted from one input image are converted to features that appear as if they were extracted from the other input image. Convert.

[0011] An exemplary embodiment of the training method includes: Multiple Extract features from the input image, multiple By the angle estimation means of Camera shooting to generate Estimate the angle, Between angles estimated by a plurality of angle estimation means Calculate the difference between the input image from The extracted features are rigidly transformed according to the difference. Extracted from the input image Calculate the matching loss between the features and the rigidly transformed features, and refer to the matching loss as at least multiple update the angle estimator of Rigid transformations make features extracted from one input image appear as if they were extracted from the other input image. will be converted. [Effects of the Invention]

[0015] According to the present invention, angle estimation can be learned without using annotations about angles or object structure. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing an example of the configuration of a training device according to a first embodiment. [Figure 2] 4 is a flowchart showing the operation of the training device of the first embodiment. [Figure 3] FIG. 1 is an explanatory diagram showing the functions and effects of the training device. [Figure 4] FIG. 10 is a block diagram showing an example of the configuration of a training device according to a second embodiment. [Figure 5] 10 is a flowchart showing the operation of the training device of the second embodiment. [Figure 6] FIG. 10 is a block diagram showing an example of the configuration of a training device according to a third embodiment. [Figure 7] 10 is a flowchart showing the operation of the training device of the third embodiment. [Figure 8] FIG. 10 is a block diagram showing an example of the configuration of a training device according to a fourth embodiment. [Figure 9] 10 is a flowchart showing the operation of the training device of the fourth embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a training device according to a fifth embodiment. [Figure 11] 10 is a flowchart showing the operation of the training device of the fifth embodiment. [Figure 12] FIG. 13 is a block diagram showing an example of the configuration of an angle estimation device according to a sixth embodiment. [Figure 13] FIG. 1 is a block diagram illustrating an example of a computer having a CPU. [Figure 14] FIG. 2 is a block diagram showing the main parts of the training device. [Figure 15] FIG. 2 is a block diagram showing the main parts of the angle estimation device. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each embodiment described below, an SAR image is assumed as the image. However, the image is not limited to an SAR image. As an example, the input image may be an optical image such as an image captured by a smartphone.

[0018] Embodiment 1. (Training equipment configuration) FIG. 1 is a block diagram showing an example of the configuration of a training device according to the first embodiment.

[0019] The training device 101 shown in FIG. 1 includes a first feature extractor 111, a second feature extractor 112, a first angle estimator 121, a second angle estimator 122, an angle difference calculation unit 130, a rigid body transformation unit 140, a matching loss calculation unit 150, and a model update unit 160.

[0020] Although one-way arrows are used in Figure 1 and other figures, the purpose of one-way arrows is to clearly represent the flow of data and does not exclude bidirectionality.

[0021] Image data I1 is input to a first feature extractor 111 and a first angle estimator 121. Image data I2 is input to a second feature extractor 112 and a second angle estimator 122. The image data I1 and I2 may be a bundle of images. An image corresponding to image data I1 is called a first image. An image corresponding to image data I2 is called a second image. The first feature extractor 111 and the second feature extractor 112 can also be configured as a single section. The first angle estimator 121 and the second angle estimator 122 can also be configured as a single section.

[0022] The relationship between the first image corresponding to input image data I1 and the second image data corresponding to input image data I2 is as follows: The second image has a different angle from the first image. As an example, the second image may be an image containing the same subject as the first image, or an image containing a different subject of the same class category, but captured at a different viewpoint (photography angle or viewing angle) than the first image. Furthermore, the first and second images may be captured at the same time or at different times.

[0023] The first feature extraction unit 111 extracts a feature (feature) f1 from the input image data I1. The second feature extraction unit 112 extracts a feature f2 from the input image data I2. The first angle estimator 121 estimates an angle θ^1 from the input image data I. The second angle estimator 122 estimates an angle θ^2 from the input image data I2. Hereinafter, θ^1 and θ^2 are referred to as estimated angles. The angle difference calculation unit 130 calculates the difference Δθ=θ^2-θ^1. Note that θ^ is equivalent to the following equation.

[0024]

number

[0025] The rigid transformation unit 140 performs rigid transformation according to Δθ on the feature f1 of the first image, and transforms the feature f1 into a novel feature f as if the new feature were extracted from an image with the same view as the second image. 1 to 2 For example, the rigid transformation unit 140 transforms the feature f1 of the input image data I1 by rotating the feature f1 on an arbitrary axis by an arbitrary angle. Note that the feature f1 may be transformed using other transformation methods as long as the transformation allows a new feature to be extracted.

[0026] The matching loss calculation unit 150 calculates the matching loss between the features extracted from the first image and the features extracted from the second image by rigid body transformation. The model update unit 160 updates the transformed novel feature f 1 to 2At least one of the trainable feature extractors 111 and 112 and the trainable angle estimators 121 and 122 is updated with reference to the matching loss so that the trainable feature f2 matches the untransformed feature f2.

[0027] (Training device operation) Next, the operation of the training device 101 will be described with reference to the flowchart of FIG.

[0028] The training device 101 receives initial model parameters (step S100). The initial model parameters include initial model parameters of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122. The received initial model parameters are supplied to the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122.

[0029] The training device 101 receives input image data I1 (step S101). The first feature extractor 111 extracts a feature f1 from the first image (step S111). The first angle estimator 121 estimates the angle of the first image (step S121). The first angle estimator 121 outputs an estimated angle θ̂1.

[0030] The training device 101 receives input image data I2 (step S102). The second feature extractor 112 extracts a feature f2 from the second image (step S112). The second angle estimator 122 estimates the angle of the second image (step S122). The second angle estimator 122 outputs an estimated angle θ^2.

[0031] The process of step S111 and the process of step S112 can be executed simultaneously. The process of step S111 and the process of step S121 can be executed simultaneously. The process of step S112 and the process of step S122 can be executed simultaneously.

[0032] The angle difference calculation unit 130 calculates the difference Δθ between the estimated angles θ1 and θ^2 (step S130). The rigid transformation unit 140 performs rigid transformation according to Δθ on the feature f1 extracted from the first image, and converts the feature f1 into a novel feature f2 as if it were extracted from an image with the same perspective as the second image. 1 to 2 (step S140).

[0033] The matching loss calculation unit 150 calculates |f 1 to 2 -f2| is calculated (step S150). The model update unit 160 determines whether the matching loss has converged (step S160). If the matching loss has converged (Yes in step S160), the process proceeds to step S162. If the matching loss has not converged (No in step S160), the process proceeds to step S161. For example, the model update unit 160 compares the matching loss with a predetermined threshold to determine whether the matching loss has converged.

[0034] In step S161, the model update unit 160 refers to the matching loss calculated by the matching loss calculation unit 150 and updates the model parameters of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122. Then, the process returns to steps S111 and S112.

[0035] In step S162, the model update unit 160 stores the model parameters of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122 in a storage medium (not shown in FIG. 1). In particular, the model parameters of the first angle estimator 121 and the second angle estimator 122 correspond to the parameters of the trained angle estimator. In other words, the model update unit 160 can output the trained angle estimator as a final processing result.

[0036] (Functions and technical effects of this embodiment) The function and effect of the training device will be described with reference to the explanatory diagram in Fig. 3. The training device of this embodiment trains an angle estimator (for example, the first angle estimator 121). Then, the trained angle estimator may estimate the angle of a newly obtained image.

[0037] In the learning stage, images of objects of the same class (e.g., image 1 shown in Figure 3, which corresponds to the first image, and image 2 shown in Figure 3, which corresponds to the second image) are provided. These images are taken at different viewing angles. As an example, image 1 taken at an angle of 0° is taken as I 0゜ It is called 、 Image 2 taken at an angle of 90° is I 90゜ (See Figure 3).

[0038] A feature extractor (e.g., a first feature extractor 111, a second feature extractor 112) extracts I 0゜ The features extracted from f 0゜ (See Figure 3) and f 90゜ The features extracted from f 90゜ Let f 0゜ and f 90゜ is shown in the middle of Figure 3 and is expressed as follows:

[0039]

number

[0040] The angle estimators (e.g., the first angle estimator 121 and the second angle estimator 122) calculate the image I 0゜ and I 90゜ As an example, assume that the estimated angle θ^1 is 20° and the estimated angle θ^2 is 65°. In this case, the difference Δθ is 45°.

[0041] The rigid body transformation part is f 0゜ As shown in the bottom right of Figure 3, the feature f transformed from 0° to 90° is expressed as follows:

[0042]

number

[0043] Transform feature f transformed from 0° to 90° view 0゜ to 90゜ and the untransformed feature f at the same 90° view 90゜ The matching loss is expressed by the following equation:

[0044]

number

[0045] If the calculation process from image input to finding the matching loss is repeated a sufficient number of times, the matching loss will likely be minimized toward zero by updating the angle estimator, or the feature extractor and angle estimator, with reference to the matching loss.

[0046] In testing or practical use, when a new image of an object of the same class is acquired, the trained angle estimator can correctly estimate its shooting angle.

[0047] Embodiment 2. (Training equipment configuration) FIG. 4 is a block diagram showing an example of the configuration of a training apparatus according to the second embodiment. The training apparatus 102 shown in FIG. 4 includes a first angle difference calculation unit 131 and a second angle difference calculation unit 132 instead of the angle difference calculation unit 130 of the first embodiment. The training apparatus 102 also includes a first rigid body conversion unit 141 and a second rigid body conversion unit 142 instead of the rigid body conversion unit 140 of the first embodiment. The other configuration of the training apparatus 102 is the same as the configuration of the training apparatus 101 of the first embodiment. The first angle difference calculation unit 131 and the second angle difference calculation unit 132 can also be configured as a single section. The first rigid body conversion unit 141 and the second rigid body conversion unit 142 can also be configured as a single section.

[0048] In this embodiment, the first angle difference calculation unit 131 and the second angle difference calculation unit 132 are provided with a reference angle θ cis input. θ c is set in advance by the user. The first angle difference calculation unit 131 calculates θ c The second angle difference calculation unit 132 calculates the difference Δθ1 between θ and θ^1. c Calculate the difference Δθ2 between θ^2 and θ^2.

[0049] The first rigid transformation unit 141 performs rigid transformation according to Δθ1 on the feature f1 extracted from the first image, and converts the feature f1 into a new feature f 1 to c The second rigid transformation unit 142 performs rigid transformation according to Δθ2 on the feature f2 extracted from the second image, and transforms the feature f2 into a new feature f 2 to c Convert to.

[0050] In this embodiment, the matching loss calculation unit 151 calculates |f 1 to c -f 2 to c Calculate |.

[0051] (Training device operation) Next, the operation of the training device 102 will be described with reference to the flowchart in Fig. 5. The processing in steps S100, S101 to S122, and S160 to S162 is the same as the processing in the training device 101 shown in Fig. 2.

[0052] In this embodiment, in step S131, the first angle difference calculation unit 131 calculates the angle θ c In step S132, the second angle difference calculation unit 132 calculates the difference Δθ1 between θ and θ. c Calculate the difference Δθ2 between θ^2 and θ^2.

[0053] The first rigid transformation unit 141 performs rigid transformation on the feature f1 according to Δθ1, and generates a new feature f1 as if the new feature had been extracted from the image of the reference view. 1 to c (Step S141). The second rigid transformation unit 142 performs rigid transformation on the feature f2 according to Δθ2, and converts it into a new feature f2 as if it were extracted from the canonical view image. 2 to c(step S142).

[0054] The matching loss calculation unit 151 calculates |f 1 to c -f 2 to c | is calculated (step S151).

[0055] (Technical effect of this embodiment) In the first embodiment, the angle difference is calculated based on the estimated angle between the first input image and the second input image, but in this embodiment, the angle difference is calculated based on, for example, the estimated angle for one of the first image or the second image and a predetermined reference angle. If the angle estimation is not yet accurate, the error in the angle difference is 2 units in the first embodiment, but the error is 1 unit in this embodiment. In other words, this embodiment is more robust than the first embodiment.

[0056] Embodiment 3. (Training equipment configuration) FIG. 6 is a block diagram showing an example of the configuration of a training device according to the third embodiment. The training device 103 shown in FIG. 6 does not include the second feature extractor 112 in the first embodiment. The training device 103 further includes a decoder 170 as an example of image reconstruction means. The other configurations of the training device 103 are the same as those of the training device 101 in the first embodiment. However, the operation of the matching loss calculation unit 152 differs from the operation of the matching loss calculation unit 150 in the first embodiment. Furthermore, the operation of the model update unit 161 differs from the operation of the model update unit 160 in the first embodiment.

[0057] The decoder 170 generates the reconstructed image Î 1 to 2 The decoder 170 also generates a reconstructed image Î1.

[0058] In this embodiment, the matching loss calculation unit 152 calculates I^ 1 to 2and I2. Furthermore, matching loss calculation unit 152 calculates the matching loss between Î1 and I1. Model update unit 161 updates at least one of trainable feature extractor 110, trainable angle estimators 121 and 122, and decoder 170 by referring to the matching loss.

[0059] (Training device operation) Next, the operation of the training device 103 will be described with reference to the flowchart in Fig. 7. The processing in steps S100, S101, S102, S111, S121, S122, S130, S140, S160, and S162 is the same as the processing in the training device 101 shown in Fig. 2.

[0060] In this embodiment, the decoder 170 converts the transformed feature f obtained from the feature f1 by the rigid body transform unit 140. 1 to 2 Using the reconstructed image I^ 1 to 2 (Step S170). The decoder 170 further generates a reconstructed image Î1 using the feature f1 in step S170.

[0061] The matching loss calculation unit 152 calculates the reconstructed image I^ 1 to 2 and image data I2, and the difference between the reconstructed image I^1 and image data I1 are calculated as matching losses (step S152). In step S161A, the model update unit 161 refers to the matching loss calculated by the matching loss calculation unit 152 and updates the model parameters of the feature extractor 110, the first angle estimator 121, the second angle estimator 122, and the decoder 170. The model update unit 161 updates the decoder 170 so that the matching loss for the reconstructed image is reduced.

[0062] (Technical effect of this embodiment) Features are a high level abstraction of images. That is, features contain much less information than images. Therefore, unlike the above embodiment, which performs comparison at the feature level, this embodiment, which performs comparison at the image level, is more robust. This is because comparing the reconstructed transformed image with the original image encourages matching of details, while comparing the transformed features with the untransformed features may ignore details but only focuses on matching contours. In other words, this embodiment is expected to be more robust than the first embodiment.

[0063] Embodiment 4. (Training equipment configuration) Fig. 8 is a block diagram showing an example of the configuration of a training device according to the fourth embodiment. The training device 104 shown in Fig. 8 further includes a first image pre-processing unit 181 and a second image pre-processing unit 182. The other configuration of the training device 104 is the same as that of the training device 101 according to the first embodiment. Note that the first image pre-processing unit 181 and the second image pre-processing unit 182 can also be configured as a single section.

[0064] The first preprocessor 181 performs predetermined preprocessing on the first image. The preprocessed image data is supplied to the first feature extractor 111 and the first angle estimator 121. The second preprocessor 182 performs predetermined preprocessing on the second image. The preprocessed image data is supplied to the second feature extractor 112 and the second angle estimator 122.

[0065] (Training device operation) Next, the operation of the training device 104 will be described with reference to the flowchart of Fig. 9. The processing in steps S100, S101 to S150, and S160 to S162 is the same as the processing in the training device 101 shown in Fig. 2.

[0066] In this embodiment, the first pre-processing unit 181 performs predetermined pre-processing on the first image in step S181. Specifically, the first pre-processing unit 181 processes image data I1. The second pre-processing unit 182 performs predetermined pre-processing on the second image in step S182. Specifically, the second pre-processing unit 182 processes image data I.

[0067] An example of preprocessing is background removal. Another example of preprocessing is noise removal. In background removal, suppose you have a photo of a vehicle driving on a road. If only the vehicle is recognized, preprocessing removes the background road. The background and the vehicle can be separated using, for example, image segmentation techniques, leaving only the image pixels of the vehicle.

[0068] Images, especially SAR images, generally contain noise. Preprocessing can be used to remove noise from optical and SAR images. Preprocessing can be performed using methods such as median filters, Gaussian blurring, fast Fourier transform-based methods, or trainable neural networks.

[0069] It should be noted that preprocessing is not limited to background removal or noise removal. It can also be designed as a trainable neural network that extracts low-level features. These low-level features are shared by the feature extractor and the angle estimator. In this way, the number of trainable parameters of the neural network can be reduced, which means that the neural network can be trained more efficiently.

[0070] (Technical effect of this embodiment) For example, by performing background removal and noise removal, the extracted features contain only or mainly object information, which allows for more accurate angle estimation.

[0071] Embodiment 5. (Training equipment configuration) Fig. 10 is a block diagram showing an example of the configuration of a training device according to the fifth embodiment. The training device 105 shown in Fig. 10 further includes a first image post-processing unit 191 and a second image post-processing unit 192. The other configuration of the training device 105 is the same as that of the training device 101 according to the first embodiment. Note that the first image post-processing unit 191 and the second image post-processing unit 192 can also be configured as a single section.

[0072] In this embodiment, the first post-processing unit 191 performs predetermined post-processing on the features extracted by the first feature extractor 111. The post-processed features are supplied to the rigid body transformation unit 140 as features f1. The second post-processing unit 192 performs predetermined post-processing on the features extracted by the second feature extractor 112. The post-processed features are supplied to the matching loss calculation unit 150 as features f2.

[0073] (Training device operation) Next, the operation of the training device 105 will be described with reference to the flowchart in Fig. 11. The processing in steps S100, S101 to S150, and S160 to S162 is the same as the processing in the training device 101 shown in Fig. 2.

[0074] In this embodiment, the first post-processing unit 191 performs predetermined post-processing on the features extracted by the first feature extractor 111. Specifically, the first post-processing unit 191 performs processing to more accurately estimate the angle. The second post-processing unit 192 performs predetermined post-processing on the features extracted by the second feature extractor 112. Specifically, the second post-processing unit 192 performs processing to more accurately estimate the angle.

[0075] An example of post-processing is normalization. Another example of post-processing is masking. When normalization is performed, the features are assumed to be, for example, a 3D point cloud. By performing point normalization, the coordinates of all points are normalized to the range of [0, 1]. Before normalization, the coordinates of some points may have very large values, such as 10, while others may have very small values, such as 0.1. This large difference significantly increases the matching loss. As a result, model training is not easy. However, in this embodiment, normalization is used to suppress unnecessary increases in matching loss.

[0076] When masking is performed, for example, if the feature is a feature map, values ​​at the boundaries are lost after rigid transformation. The masking filter retains only the values ​​at the center. In this embodiment, masking is performed to make the transformed features with lost values ​​at the boundaries equivalent to features that have not been transformed at the boundaries. Note that post-processing is not limited to normalization or masking, and may also include trainable neural networks such as conditional generative networks.

[0077] (Technical effect of this embodiment) By normalizing the points and masking the feature maps, the features (including transformed and untransformed features) become more suitable for comparison, which leads to more accurate angle estimation.

[0078] Embodiment 6. Fig. 12 is a block diagram showing an example of the configuration of an angle estimation device according to the sixth embodiment. The angle estimation device 201 shown in Fig. 12 includes an angle estimator 61. The angle estimator 61 is a device that has been trained by the training device 100.

[0079] The training device 100 corresponds to any one of the training devices 101 to 105 of the first to fifth embodiments. The angle estimator 61 corresponds to, for example, the first angle estimator 121 or the second angle estimator 122 extracted from any one of the training devices 101 to 105 of the first to fifth embodiments.

[0080] Therefore, the angle estimator 61 can be trained by the training device 100 as described in the first to fifth embodiments.

[0081] The angle estimator 61 of this embodiment can correctly estimate the angle information of the input image.

[0082] Each component in the above embodiment can be configured as a single piece of hardware, or as a single piece of software. Each component can also be configured as multiple pieces of hardware, or as multiple pieces of software. Furthermore, some of the components can be configured as hardware, and the other parts can be configured as software.

[0083] Each function (each process) in the above-described embodiments can be realized by a computer having a processor such as a CPU (Central Processing Unit), a memory, etc. For example, a program for implementing the method (process) in the above-described embodiments may be stored in a storage device (storage medium), and each function may be realized by executing the program stored in the storage device by a CPU.

[0084] 13 is a block diagram showing an example of a computer having a CPU. The computer is implemented in the training device and the classification device. The CPU 1000 executes processing in accordance with a program stored in a storage device 1001 to realize the functions of the above-described embodiment. That is, by executing the program stored in the storage device 1001, the computer can realize the functions of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, the second angle estimator 122, the angle difference calculation unit 130, the first angle difference calculation unit 131, the second angle difference calculation unit 132, the rigid body transformation unit 140, the first rigid body transformation unit 141, the second rigid body transformation unit 142, the matching loss calculation units 150, 151, 152, the model update units 160, 161, the decoder 170, the first image pre-processing unit 181, the second image pre-processing unit 182, the first image post-processing unit 191, and the second image post-processing unit 192 in the training apparatus shown in Figures 1, 4, 6, 8, and 10.

[0085] The computer can implement the functions of the angle estimator 61 in the angle estimation device shown in FIG. 12 by executing a program stored in the storage device 1001.

[0086] The storage device 1001 is, for example, a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage medium. Specific examples of non-transitory computer readable media include magnetic recording media (e.g., hard disks), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Compact Disc-Read Only Memory), CD-Rs (Compact Disc-Recordable), CD-R / Ws (Compact Disc-ReWritable), and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), and flash ROMs).

[0087] The program may also be stored in various types of temporary computer-readable media, to which the program is supplied, for example, via a wired or wireless communication path, or via an electrical signal, an optical signal, or an electromagnetic wave.

[0088] The memory 1002 is realized by, for example, a random access memory (RAM), and is a storage means for temporarily storing data when the CPU 1000 executes processing. A configuration is also conceivable in which a program held in the storage device 1001 or a temporary computer-readable medium is transferred to the memory 1002, and the CPU 1000 executes processing based on the program in the memory 1002.

[0089] FIG. 14 is a block diagram showing the main parts of a training device. The training device 10 shown in FIG. 14 includes one or more feature extraction means 11 (in the embodiment, realized by a first feature extractor 111 and a second feature extractor 112), one or more angle estimation means 12 (in the embodiment, realized by a first angle estimator 121 and a second angle estimator 122) that estimate an angle from an input image, angle difference calculation means 13 (in the embodiment, realized by an angle difference calculation unit 130) that calculates an angle difference, angle difference calculation means 13 (in the embodiment, realized by an angle difference calculation unit 130, or a first angle difference calculation unit 131 and a second angle difference calculation unit 132) that calculates the difference between the angles estimated by the one or more angle estimation means, and rigid body transformation means 14 (in the embodiment, realized by a rigid body transformation unit 140, or a first rigid body transformation unit 141) that transforms the features of the input image in accordance with the difference. The rigid body transformation unit 14 transforms the non-transformed features so that they appear as if they were extracted from an image at the same angle as the image from which they were extracted.

[0090] Fig. 15 is a block diagram showing the main parts of an angle estimation device. An angle estimation device 20 shown in Fig. 15 includes angle estimation means 21 (implemented by an angle estimator 61 in this embodiment) that estimates an angle from an input image, and angle estimation means 21 has been trained together with one or more feature extraction means and rigid body transformation means so that extracted image features transformed according to the estimated angle appear as if they were extracted from an image at a different angle from the original input image.

[0091] In each of the above embodiments, the image is typically a SAR image. However, the image is not limited to a SAR image. For example, the image can also be an optical image, such as an image taken by a smartphone.

[0092] Because the trained angle estimation device of the above embodiment can correctly estimate angle information of an image, the trained angle estimation device can be incorporated into other image processing systems to provide angle information and improve the overall performance of the system. For example, if the angle estimation device of the above embodiment provides an estimated head pose of a human face image to a face recognition system, the system's recognition accuracy will be improved because it has special knowledge about human head pose.

[0093] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0094] (Supplementary Note 1) One or more feature extraction means for extracting features from an input image; one or more angle estimation means for estimating angles from an input image; An angle difference calculation means for calculating a difference between the angles estimated by the one or more angle estimation means; rigid body transformation means for transforming the features of the input image in accordance with the difference; a matching loss calculation means for calculating a matching loss between the untransformed features extracted by the one or more feature extraction means and the features transformed by the rigid body transformation means; updating means for updating at least one or more angle estimators by referring to the matching loss; The rigid transformation means transforms the features so that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted. Training equipment.

[0095] (Supplementary Note 2) The updating means also updates the one or more feature extraction means. 10. The training device of claim 1.

[0096] (Supplementary Note 3) The device further includes one or more preprocessing means for performing predetermined preprocessing on the input image and supplying the preprocessed image to the one or more feature extraction means and the one or more angle estimation means. 10. The training device of claim 1 or 2.

[0097] (Supplementary Note 4) The system further includes one or more post-processing means for performing predetermined post-processing on the features extracted by the one or more feature extraction means and the features output by the rigid body transformation means. 10. The training device of claim 1 or 2.

[0098] (Appendix 5) One or more feature extraction means for extracting features from an input image; one or more angle estimation means for estimating angles from an input image; one or more angle difference calculation means for calculating a difference between the angle estimated by the one or more angle estimation means and a reference view angle; one or more rigid body transformation means for transforming the non-transformed features extracted by the one or more feature extraction means in accordance with the difference; a matching loss calculation means for calculating a matching loss between features transformed by the one or more rigid body transformation means; updating means for updating at least the one or more angle estimation means and the one or more feature extraction means by referring to the matching loss; The rigid transformation means transforms the features so that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted. Training equipment.

[0099] (Appendix 6) A feature extraction means for extracting features from an input image; one or more angle estimation means for estimating angles from an input image; An angle difference calculation means for calculating a difference between the angles estimated by the one or more angle estimation means; rigid body transformation means for transforming the features of the input image in accordance with the difference; an image reconstruction means for reconstructing an image using the features extracted by the feature extraction means and an image using the features rigidly transformed by the rigid transformation means; a matching loss calculation means for calculating a matching loss between an image reconstructed from the transformed features and an input image at the same angle as the angle at which the features are transformed, and between an image reconstructed from the features without rigid body transformation and the input image from which the features are extracted; an update unit that updates at least one of the angle estimation unit, the feature extraction unit, and the image reconstruction unit by referring to the matching loss; The rigid transformation means transforms the feature so that it appears as if it were extracted from an image at the same angle as the image from which the feature was extracted. Training equipment.

[0100] (Supplementary Note 7) The matching loss calculation means repeatedly calculates the matching loss until the matching loss converges. 7. The training device of any one of claims 1 to 6.

[0101] (Appendix 8) An angle estimation means for estimating an angle from an input image is provided, The angle estimation means, together with one or more feature extraction means and rigid body transformation means, is trained to make the image features appear as if they were extracted from an image at a different angle than the original input image after being transformed according to the estimated angle. Angle estimation device.

[0102] (Supplementary Note 9) A training method for training a device having one or more angle estimators, comprising: Extract features from the input image, estimating angles from the input images by one or more angle estimators; Calculate the difference in estimated angles, The extracted features of the input image are rigidly transformed according to the difference; Compute the matching loss between the untransformed features and the rigidly transformed features, updating at least one or more angle estimators with reference to the matching loss; The transformed features are transformed so that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted. Training methods.

[0103] (Appendix 10) Training a device having one or more feature extraction means; The one or more feature extraction means are updated with reference to the matching loss. 10. The training method described in Appendix 9.

[0104] (Appendix 11) Estimating angles from an input image using an angle estimation device that is trained together with one or more feature extraction means and rigid transformation means so that, after being transformed according to the estimated angle, the image features appear as if they were extracted from an image at a different angle from the original input image. Angle estimation method.

[0105] (Supplementary Note 12) A computer-readable information recording medium storing a training program for training a device having one or more angle estimation means, the training program being configured to instruct a computer to: Extract features from the input image, causing one or more angle estimators to estimate angles from the input images; Calculate the difference in estimated angles, The extracted features of the input image are rigidly transformed according to the difference; Compute the matching loss between the untransformed features and the rigidly transformed features, updating at least one or more angle estimators with reference to the matching loss; The transformed features are transformed so that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted.

[0106] (Supplementary Note 13) The training program is a program for executing one or more feature extraction means for extracting features from an input image on a computer. The one or more feature extraction methods are updated with reference to the matching loss. 13. The computer-readable storage medium of claim 12.

[0107] (Supplementary Note 14) A computer-readable information recording medium having recorded thereon an angle estimation program, the angle estimation program being configured to execute the following in a computer: An angle is estimated from an input image using an angle estimation device that has been trained to use one or more feature extraction means and rigid transformation means so that, after being transformed according to the estimated angle, the image features appear as if they were extracted from an image at a different angle than the original input image.

[0108] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0109] 10,101~105 Training equipment 11 Feature extraction methods 12 Angle estimation means 13 Angle difference calculation means 14 Rigid body transformation means 15 Matching loss calculation method 16 Update method 20,201 Angle estimation device 21 Angle estimation means 61 Angle Estimator 100 Training Equipment 111 First feature extractor 112 Second feature extractor 121 First angle estimator 122 Second angle estimator 130 Angle Difference Calculation Section 131 First angle difference calculation unit 132 Second angle difference calculation unit 140 Rigid body transformation part 141 First rigid body transformation part 142 Second rigid body transformation part 150, 151, 152 Matching loss calculation section 160,161 Model Update Section 170 Decoder 181 First Image Preprocessor 182 Second Image Preprocessor 191 First Image Post-Processor 192 Second Image Post-Processor

Claims

1. A method for detecting features from a plurality of input images, comprising: a plurality of angle estimation means for estimating the shooting angle of a camera that generates the input image; An angle difference calculation means for calculating a difference between angles estimated by the plurality of angle estimation means; rigid transformation means for transforming the features extracted from the input image in accordance with the difference; a matching loss calculation means for calculating a matching loss between a feature extracted by any one of the feature extraction means and a feature transformed by the rigid body transformation means; updating means for updating at least the plurality of angle estimators by referring to the matching loss; The rigid transformation means transforms a feature extracted from one of the input images into a feature that appears as if it were extracted from another of the input images. Training equipment.

2. The updating means also updates the plurality of feature extracting means.

10. The training device of claim 1.

3. The image processing device further includes a plurality of post-processing means for performing normalization, masking, or post-processing using a trainable neural network on the features extracted by the plurality of feature extraction means.

3. The training device according to claim 1 or 2.

4. A method for detecting features from a plurality of input images, comprising: a plurality of angle estimation means for estimating the shooting angle of a camera that generates the input image; a plurality of angle difference calculation means for calculating the difference between the angles estimated by the plurality of angle estimation means and a reference angle of view; a plurality of rigid body transformation means for transforming the features extracted by the plurality of feature extraction means in accordance with the difference; a matching loss calculation means for calculating a matching loss between the features transformed by the plurality of rigid body transformation means; an update unit that updates at least the plurality of angle estimation units and the plurality of feature extraction units by referring to the matching loss; The rigid body transformation means transforms each feature extracted by the plurality of feature extraction means into a feature that appears to have been extracted from the image of the reference view. Training equipment.

5. A feature extraction means for extracting features from a plurality of input images; a plurality of angle estimation means for estimating the shooting angle of a camera that generates the input image; An angle difference calculation means for calculating a difference between angles estimated by the plurality of angle estimation means; rigid transformation means for transforming the features extracted from the input image in accordance with the difference; an image reconstruction means for reconstructing an image using the features extracted by the feature extraction means and an image using the features rigidly transformed by the rigid transformation means; a matching loss calculation means for calculating a matching loss between an image reconstructed from the transformed features and an input image at the same angle as the angle at which the features are transformed, and between an image reconstructed from the features without rigid body transformation and the input image from which the features are extracted; an update unit that updates at least one of the angle estimation unit, the feature extraction unit, and the image reconstruction unit by referring to the matching loss; The rigid transformation means transforms a feature extracted from one of the input images into a feature that appears as if it were extracted from another of the input images. Training equipment.

6. The matching loss calculation means repeatedly calculates the matching loss until the matching loss converges. A training device according to any one of claims 1 to 5.

7. A training method for training a device having a plurality of angle estimation means, comprising: Extract features from multiple input images, Estimating the shooting angle of a camera that generates the input image by the plurality of angle estimation means; calculating a difference between the angles estimated by the plurality of angle estimation means; The features extracted from the input image are rigidly transformed according to the difference. Calculating a matching loss between the features extracted from the input image and the rigidly transformed features; updating at least the plurality of angle estimators with reference to a matching loss; The rigid transformation transforms features extracted from any of the input images into features that appear as if they were extracted from any other of the input images. Training methods.

Images