Training device, angle estimation device, training method, and angle estimation method

Abstract

The training device includes an angle difference calculation unit that calculates a difference between angles estimated by one or more angle estimators, a rigid transformation unit that transforms features of an input image according to the difference, a matching loss calculation unit that calculates a matching loss between non-transformed features extracted by one or more feature extractors and features transformed by the rigid transformation unit, and an update unit that updates at least one or more angle estimators by referring to the matching loss, and the rigid transformation unit transforms the features so that they appear as if they were extracted from an image with the same angle as the image from which the non-transformed features were extracted.

Description

[Technical field]

[0001] The present invention relates to a training device, an angle estimation device, a training method, and an angle estimation method. [Background technology]

[0002] Angular information is useful for training object classifiers in SAR (synthetic aperture radar), especially when data is limited. Angular information is also useful for tasks related to optical imagery, such as face and object recognition.

[0003] An angle estimator can estimate angle information from an input image. The angle information includes, but is not limited to, the pose of an object in an image and the shooting angle of the camera that generates the image. The angle information helps improve the performance of a classifier. For example, a classifier trained on only two images, A and B, without knowing the angle information, will only work on images with angles A and B. However, a classifier trained on the same images with the angle information will work on images with angles other than A and B by interpolating the image information between angles A and B. In face and object recognition, with angle information, it is possible to infer how an object looks when viewed from a new angle, even if there is no training data for the new angle.

[0004] To improve the accuracy of the angle estimator, the angle estimator is trained by machine learning. There is a method of using a correct angle label to train the angle estimator (see, for example, Non-Patent Document 1). In this method, an image and its correct image label are first input. Next, the angle estimator, which is a trainable neural network, estimates an angle from the image. Furthermore, the angle estimator is updated according to a penalty. After thousands of repeated updates, the angle estimator estimates an angle that matches the correct angle label.

[0005] There is also a method that uses ground-truth structure data (see, for example, Non-Patent Document 2). In this method, an image and its ground-truth structure data are first input. Next, features are extracted from a two-dimensional (2D) image using a trainable feature extractor. Furthermore, an angle is estimated from the 2D image using a trainable angle extractor. Furthermore, the structure data is projected according to the estimated angle to obtain a projected feature. Then, a penalty is calculated as the difference between the extracted feature and the projected feature. After that, the feature extractor and the angle estimator are updated according to the penalty. After thousands of repeated updates, the angle estimator estimates the angle so that the projected feature matches the extracted feature. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] S Tulsiani et al., "Viewpoints and keypoints", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1510-1519 [Non-Patent Document 2] G Pavlakos et al., "6-dof object pose from semantic keypoints", May 2017, In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2011-2018 Summary of the Invention [Problem to be solved by the invention]

[0007] The method using ground-truth angle labels requires guidance from angle labels. The method using ground-truth structure data requires guidance from structure data. It is desirable to train an angle estimator without guidance from angle labels, structure data, etc.

[0008] Therefore, an object of the present invention is to provide a training device, an angle estimation device, a training method, and an angle estimation method that are capable of learning a model related to angle estimation without using annotations. [Means for solving the problem]

[0009] An exemplary embodiment of the training apparatus includes one or more feature extraction means for extracting features from an input image, one or more angle estimation means for estimating angles from the input image, an angle difference calculation means for calculating a difference between angles estimated by the one or more angle estimation means, a rigid transformation means for transforming features of the input image according to the difference, a matching loss calculation means for calculating a matching loss between untransformed features extracted by the one or more feature extraction means and features transformed by the rigid transformation means, and an update means for updating at least one or more angle estimation means with reference to the matching loss, wherein the rigid transformation means transforms the features so that they appear as if they were extracted from an image with the same angle as the image from which the untransformed features were extracted.

[0010] An exemplary embodiment of an angle estimation apparatus includes an angle estimation means for estimating an angle from an input image, where the angle estimation means, together with one or more feature extraction means and rigid body transformation means, is trained to make the image features appear as if they were extracted from an image at a different angle than the original input image after the angle estimation means is transformed according to the estimated angle.

[0011] An exemplary embodiment of the training method includes extracting features from an input image, estimating angles from the input image by one or more angle estimators, calculating a difference in the estimated angles, rigidly transforming the extracted features of the input image according to the difference, calculating a matching loss between the untransformed features and the rigidly transformed features, updating at least one or more angle estimators with reference to the matching loss, and transforming the transformed features such that they appear as if they were extracted from an image with the same angle as the image from which the untransformed features were extracted.

[0012] An exemplary embodiment of the angle estimation method estimates an angle from an input image using an angle estimator that is trained in conjunction with one or more feature extraction means and rigid transformation means such that image features, after being transformed according to the estimated angle, appear as if they were extracted from an image at a different angle than the original input image.

[0013] An exemplary embodiment of the training program causes a computer to extract features from an input image, estimate angles from the input image by one or more angle estimators, calculate a difference between the estimated angles, rigidly transform the extracted features of the input image according to the difference, calculate a matching loss between the untransformed features and the rigidly transformed features, update at least one or more angle estimators with reference to the matching loss, and transform the transformed features such that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted.

[0014] An exemplary embodiment of the angle estimation program causes a computer to estimate an angle from an input image using an angle estimator that has been trained, along with one or more feature extraction means and rigid body transformation means, such that image features, after being transformed according to the estimated angle, appear as if they were extracted from an image at a different angle than the original input image. Effect of the Invention

[0015] We are able to learn angle estimation without using annotations on angles or object structure. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing an example of the configuration of a training device according to a first embodiment. [Diagram 2] 4 is a flowchart showing the operation of the training device of the first embodiment. [Diagram 3] FIG. 2 is an explanatory diagram showing the functions and effects of the training device. [Figure 4]FIG. 13 is a block diagram showing an example of the configuration of a training device according to a second embodiment. [Diagram 5] 10 is a flowchart showing the operation of the training device of the second embodiment. [Figure 6] FIG. 13 is a block diagram showing an example of the configuration of a training device according to a third embodiment. [Figure 7] 10 is a flowchart showing the operation of the training device of the third embodiment. [Figure 8] FIG. 13 is a block diagram showing an example of the configuration of a training device according to a fourth embodiment. [Figure 9] 13 is a flowchart showing the operation of the training device of the fourth embodiment. [Figure 10] FIG. 13 is a block diagram showing an example of the configuration of a training device according to a fifth embodiment. [Figure 11] 13 is a flowchart showing the operation of the training device of the fifth embodiment. [Figure 12] FIG. 13 is a block diagram showing an example of the configuration of an angle estimation device according to a sixth embodiment. [Figure 13] FIG. 1 is a block diagram showing an example of a computer having a CPU. [Figure 14] FIG. 2 is a block diagram showing the main parts of the training device. [Figure 15] 1 is a block diagram showing a main part of an angle estimation device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each embodiment described below, an SAR image is assumed as the image. However, the image is not limited to an SAR image. As an example, the input image may be an optical image such as an image taken by a smartphone.

[0018] Embodiment 1. (Training Equipment Configuration) FIG. 1 is a block diagram showing an example of the configuration of a training device according to the first embodiment.

[0019] The training device 101 shown in FIG. 1 includes a first feature extractor 111, a second feature extractor 112, a first angle estimator 121, a second angle estimator 122, an angle difference calculation unit 130, a rigid body transformation unit 140, a matching loss calculation unit 150, and a model update unit 160.

[0020] Although one-way arrows are used in Figure 1 and other figures, these are intended to make the flow of data easier to understand and do not exclude bidirectionality.

[0021] Image data I1 is input to a first feature extractor 111 and a first angle estimator 121. Image data I2 is input to a second feature extractor 112 and a second angle estimator 122. Image data I1 and I2 may be a bundle of images. An image corresponding to image data I1 is called a first image. Also, an image corresponding to image data I2 is called a second image. Note that the first feature extractor 111 and the second feature extractor 112 can be configured as one section. Also, the first angle estimator 121 and the second angle estimator 122 can be configured as one section.

[0022] The relationship between the first image corresponding to the input image data I1 and the second image data corresponding to the input image data I2 is as follows. The second image has a different angle from the first image. As an example, the second image may be an image including the same object as the first image, or an image including a different object of the same class category, but taken at a different viewpoint (photographing angle or viewing angle) than the first image. In addition, the first image and the second image may be taken at the same time, or at different times.

[0023] The first feature extraction unit 111 extracts a feature amount (feature) f1 from the input image data I1. The second feature extraction unit 112 extracts a feature f2 from the input image data I2. The first angle estimator 121 estimates an angle θ^1 from the input image data I. The second angle estimator 122 estimates an angle θ^2 from the input image data I2. Hereinafter, θ^1 and θ^2 are referred to as estimated angles. The angle difference calculation unit 130 calculates the difference Δθ=θ^2-θ^1. Note that θ^ is equivalent to the following equation.

[0024]

number

[0025] The rigid transformation unit 140 performs rigid transformation according to Δθ on the feature f1 of the first image, and transforms the feature f1 into a novel feature f as if the new feature was extracted from an image with the same view as the second image. 1 to 2 For example, the rigid transformation unit 140 transforms the feature f1 of the input image data I1 by rotating the feature f1 on an arbitrary axis by an arbitrary angle. Note that the feature f1 may be transformed using other transformation methods as long as the transformation is performed so that a new feature is extracted.

[0026] The matching loss calculation unit 150 calculates a matching loss between the feature extracted from the first image by the rigid body transformation and the feature extracted from the second image. The model update unit 160 updates the transformed novel feature f 1 to 2 At least one of the trainable feature extractors 111 and 112 and the trainable angle estimators 121 and 122 is updated with reference to the matching loss so that the trainable feature f2 matches the untransformed feature f2.

[0027] (Training device operation) Next, the operation of the training device 101 will be described with reference to the flow chart of FIG.

[0028] The training device 101 receives initial model parameters (step S100). The initial model parameters include initial model parameters of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122. The received initial model parameters are provided to the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122.

[0029] The training device 101 receives input image data I1 (step S101). The first feature extractor 111 extracts a feature f1 from the first image (step S111). The first angle estimator 121 estimates the angle of the first image (step S121). The first angle estimator 121 outputs an estimated angle θ̂1.

[0030] The training device 101 receives input image data I2 (step S102). The second feature extractor 112 extracts a feature f2 from the second image (step S112). The second angle estimator 122 estimates the angle of the second image (step S122). The second angle estimator 122 outputs an estimated angle θ^2.

[0031] The process of step S111 and the process of step S112 can be executed simultaneously. The process of step S111 and the process of step S121 can be executed simultaneously. The process of step S112 and the process of step S122 can be executed simultaneously.

[0032] The angle difference calculation unit 130 calculates the difference Δθ between the estimated angles θ1 and θ^2 (step S130). The rigid body transformation unit 140 performs rigid body transformation according to Δθ on the feature f1 extracted from the first image, and converts the feature f1 into a novel feature f2 as if it were extracted from an image with the same perspective as the second image. 1 to 2 (step S140).

[0033] The matching loss calculation unit 150 calculates the matching loss |f 1 to 2-f2| is calculated (step S150). The model update unit 160 determines whether the matching loss has converged (step S160). If the matching loss has converged (Yes in step S160), the process proceeds to step S162. If the matching loss has not converged (No in step S160), the process proceeds to step S161. For example, the model update unit 160 compares the matching loss with a predetermined threshold value to determine whether the matching loss has converged.

[0034] In step S161, the model update unit 160 refers to the matching loss calculated by the matching loss calculation unit 150, and updates the model parameters of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122. Then, the process returns to steps S111 and S112.

[0035] In step S162, the model update unit 160 stores model parameters of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, and the second angle estimator 122 in a storage medium (not shown in FIG. 1). In particular, the model parameters of the first angle estimator 121 and the second angle estimator 122 correspond to the parameters of the trained angle estimator. In other words, the model update unit 160 can output the trained angle estimator as a final processing result.

[0036] (Functions and technical effects of this embodiment) The function and effect of the training device will be described with reference to the explanatory diagram of Fig. 3. The training device of this embodiment trains an angle estimator (for example, the first angle estimator 121). Then, the trained angle estimator may estimate the angle of a newly obtained image.

[0037] In the learning stage, images of objects of the same class (e.g., image 1 shown in FIG. 3, which corresponds to the first image, and image 2 shown in FIG. 3, which corresponds to the second image) are provided. These images are taken at different viewing angles. As an example, image 1 taken at an angle of 0° is taken as I 0゜ Call it 、Image 2 taken at an angle of 90° is I 90゜ (See Figure 3.)

[0038] A feature extractor (e.g., a first feature extractor 111, a second feature extractor 112) extracts I 0゜ The features extracted from f 0゜ (See Figure 3) and f 90゜ The features extracted from f 90゜ Let f 0゜ and f 90゜ is shown in the middle part of FIG. 3 and expressed as follows:

[0039]

number

[0040] The angle estimator (e.g., the first angle estimator 121 and the second angle estimator 122) calculates the image I 0゜ and I 90゜ As an example, assume that the estimated angle θ^1 is 20° and the estimated angle θ^2 is 65°. In this case, the difference Δθ is 45°.

[0041] The rigid body transformation part is f 0゜ As shown in the lower right of Figure 3, the feature f transformed from 0° to 90° is expressed as follows:

[0042]

number

[0043] Transformation feature f transformed from 0° to 90° of view 0゜ to 90゜ and the untransformed feature f at the same 90° view 90゜ The matching loss with is expressed by the following equation:

[0044]

number

[0045] If the calculation process from image input to calculating the matching loss is repeated a sufficient number of times, the matching loss will likely be minimized toward zero by updating the angle estimator, or the feature extractor and angle estimator, with reference to the matching loss.

[0046] In the testing or practical application phase, when we acquire a new image of an object of the same class, the trained angle estimator is able to correctly estimate its shooting angle.

[0047] Embodiment 2. (Training Equipment Configuration) FIG. 4 is a block diagram showing an example of the configuration of a training device of the second embodiment. The training device 102 shown in FIG. 4 includes a first angle difference calculation unit 131 and a second angle difference calculation unit 132 instead of the angle difference calculation unit 130 of the first embodiment. In addition, the training device 102 includes a first rigid body conversion unit 141 and a second rigid body conversion unit 142 instead of the rigid body conversion unit 140 of the first embodiment. The other configurations of the training device 102 are the same as those of the training device 101 of the first embodiment. Note that the first angle difference calculation unit 131 and the second angle difference calculation unit 132 can also be configured as one section. In addition, the first rigid body conversion unit 141 and the second rigid body conversion unit 142 can also be configured as one section.

[0048] In this embodiment, the first angle difference calculation unit 131 and the second angle difference calculation unit 132 are provided with a reference angle θ c is input. θ c is set in advance by the user. The first angle difference calculation unit 131 calculates θ c The second angle difference calculation unit 132 calculates the difference Δθ1 between θ c Calculate the difference Δθ2 between θ^2.

[0049] The first rigid transformation unit 141 performs rigid transformation according to Δθ1 on the feature f1 extracted from the first image to convert the feature f1 into a new feature f 1 to cThe second rigid transformation unit 142 performs rigid transformation according to Δθ2 on the feature f2 extracted from the second image, and transforms the feature f2 into a new feature f 2 to c Convert to.

[0050] In this embodiment, the matching loss calculation unit 151 calculates the matching loss by |f 1 to c -f 2 to c Calculate |.

[0051] (Training device operation) Next, the operation of the training device 102 will be described with reference to the flowchart in Fig. 5. The processes in steps S100, S101 to S122, and S160 to S162 are the same as those of the training device 101 shown in Fig. 2.

[0052] In this embodiment, in step S131, the first angle difference calculation unit 131 calculates the angle θ c In step S132, the second angle difference calculation unit 132 calculates the difference Δθ1 between θ c Calculate the difference Δθ2 between θ^2.

[0053] The first rigid transformation unit 141 performs rigid transformation on the feature f1 according to Δθ1, and converts the feature f1 into a new feature f2 as if the new feature had been extracted from the image of the reference view. 1 to c (Step S141). The second rigid transformation unit 142 performs rigid transformation on the feature f2 according to Δθ2, and converts it into a new feature f2 as if it were extracted from the canonical view image. 2 to c (step S142).

[0054] The matching loss calculation unit 151 calculates the matching loss |f 1 to c -f 2 to c | is calculated (step S151).

[0055] (Technical effect of this embodiment) In the first embodiment, the angle difference is calculated based on the estimated angle for the first input image and the second input image, but in this embodiment, the angle difference is calculated based on, for example, the estimated angle for one of the first image or the second image and a predetermined reference angle. If the angle estimation is not yet accurate, the error of the angle difference is 2 units in the first embodiment, but the error is 1 unit in this embodiment. That is, this embodiment is more robust than the first embodiment.

[0056] Embodiment 3. (Training Equipment Configuration) FIG. 6 is a block diagram showing an example of the configuration of a training device of the third embodiment. The training device 103 shown in FIG. 6 does not include the second feature extractor 112 in the first embodiment. In addition, the training device 103 further includes a decoder 170 as an example of an image reconstruction means. Other configurations of the training device 103 are the same as those of the training device 101 in the first embodiment. However, the operation of the matching loss calculation unit 152 is different from the operation of the matching loss calculation unit 150 in the first embodiment. In addition, the operation of the model update unit 161 is different from the operation of the model update unit 160 in the first embodiment.

[0057] The decoder 170 outputs the reconstructed image Î 1 to 2 The decoder 170 also generates a reconstructed image Î1.

[0058] In this embodiment, the matching loss calculation unit 152 calculates Î 1 to 2 and I2. Furthermore, matching loss calculation unit 152 calculates the matching loss between Î1 and I1. Model update unit 161 updates at least one of trainable feature extractor 110, trainable angle estimators 121 and 122, and decoder 170 by referring to the matching loss.

[0059] (Training device operation) Next, the operation of the training device 103 will be described with reference to the flowchart of Fig. 7. The processes of steps S100, S101, S102, S111, S121, S122, S130, S140, S160, and S162 are the same as those of the training device 101 shown in Fig. 2.

[0060] In this embodiment, the decoder 170 converts the feature f1 obtained by the rigid body transform unit 140 into the converted feature f2. 1 to 2 Using the reconstructed image I^ 1 to 2 (Step S170). The decoder 170 further generates a reconstructed image Î1 using the feature f1 in step S170.

[0061] The matching loss calculation unit 152 calculates the reconstructed image I^ 1 to 2 and image data I2, and the difference between the reconstructed image I^1 and image data I1 are calculated as matching losses (step S152). In step S161A, the model update unit 161 refers to the matching loss calculated by the matching loss calculation unit 152, and updates model parameters of the feature extractor 110, the first angle estimator 121, the second angle estimator 122, and the decoder 170. The model update unit 161 updates the decoder 170 so that the matching loss for the reconstructed image is reduced.

[0062] (Technical effect of this embodiment) Features are a high level abstraction of images. That is, features contain much less information than images. Therefore, unlike the above embodiment, which performs comparison at the feature level, this embodiment, which performs comparison at the image level, is more robust because comparing the reconstructed transformed image with the original image encourages matching of details, whereas comparing the transformed features with the untransformed features may ignore details but only focuses on matching contours. In other words, this embodiment is expected to be more robust than the first embodiment.

[0063] Embodiment 4. (Training Equipment Configuration) Fig. 8 is a block diagram showing an example of the configuration of a training device according to the fourth embodiment. The training device 104 shown in Fig. 8 further includes a first image pre-processing unit 181 and a second image pre-processing unit 182. The other configurations of the training device 104 are the same as those of the training device 101 according to the first embodiment. Note that the first image pre-processing unit 181 and the second image pre-processing unit 182 can also be configured as one section.

[0064] The first preprocessor 181 performs a predetermined preprocessing on the first image. The preprocessed image data is supplied to the first feature extractor 111 and the first angle estimator 121. The second preprocessor 182 performs a predetermined preprocessing on the second image. The preprocessed image data is supplied to the second feature extractor 112 and the second angle estimator 122.

[0065] (Training device operation) Next, the operation of the training device 104 will be described with reference to the flowchart of Fig. 9. The processes of steps S100, S101 to S150, and S160 to S162 are the same as those of the training device 101 shown in Fig. 2.

[0066] In this embodiment, the first pre-processing unit 181 performs a predetermined pre-processing on the first image in step S181. Specifically, the first pre-processing unit 181 processes image data I1. The second pre-processing unit 182 performs a predetermined pre-processing on the second image in step S182. Specifically, the second pre-processing unit 182 processes image data I.

[0067] One example of pre-processing is background removal. Another example of pre-processing is noise removal. In background removal, assume that a photo of a vehicle driving on a road is obtained. If only the vehicle is recognized, the pre-processing removes the background, which is the road. The background and the vehicle can be separated, for example, using image segmentation techniques, and only the image pixels of the vehicle remain.

[0068] In general, images, especially SAR images, contain noise. For denoising, pre-processing can be used to remove noise from optical and SAR images. Pre-processing can be done using median filters, Gaussian blurring, methods based on the fast Fourier transform, or trainable neural networks.

[0069] It should be noted that pre-processing is not limited to background removal or noise removal. Pre-processing can also be designed as a trainable neural network that extracts low-level features. These low-level features are shared by the feature extractor and the angle estimator. In this way, the number of trainable parameters of the neural network can be reduced, which means that the neural network can be trained more efficiently.

[0070] (Technical effect of this embodiment) For example, by performing background removal and noise removal, the extracted features contain only or mainly object information, which allows for more accurate angle estimation.

[0071] Embodiment 5. (Training Equipment Configuration) Fig. 10 is a block diagram showing an example of the configuration of a training device according to the fifth embodiment. The training device 105 shown in Fig. 10 further includes a first image post-processing unit 191 and a second image post-processing unit 192. The other configurations of the training device 105 are the same as those of the training device 101 according to the first embodiment. Note that the first image post-processing unit 191 and the second image post-processing unit 192 can also be configured as one section.

[0072] In this embodiment, the first post-processing unit 191 performs a predetermined post-processing on the features extracted by the first feature extractor 111. The post-processed features are supplied to the rigid body transformation unit 140 as features f1. The second post-processing unit 192 performs a predetermined post-processing on the features extracted by the second feature extractor 112. The post-processed features are supplied to the matching loss calculation unit 150 as features f2.

[0073] (Training device operation) Next, the operation of the training device 105 will be described with reference to the flowchart in Fig. 11. The processes in steps S100, S101 to S150, and S160 to S162 are the same as those in the training device 101 shown in Fig. 2.

[0074] In this embodiment, the first post-processing unit 191 performs a predetermined post-processing on the features extracted by the first feature extractor 111. Specifically, the first post-processing unit 191 performs a process for more accurate angle estimation. The second post-processing unit 192 performs a predetermined post-processing on the features extracted by the second feature extractor 112. Specifically, the second post-processing unit 192 performs a process for more accurate angle estimation.

[0075] An example of post-processing is normalization. Another example of post-processing is masking. When normalization is performed, the features are assumed to be, for example, a 3D point cloud. By performing point normalization, the coordinates of all points are normalized in the range of [0,1]. Before normalization, the coordinates of some points may have a very large value, for example 10, while they may also have a very small value, for example 0.1, and this large difference makes the matching loss very large. As a result, it is not easy to learn the model. However, in this embodiment, unnecessary increases in the matching loss are suppressed by normalization.

[0076] When masking is performed, for example, if the feature is a feature map, the boundary values ​​are lost after rigid body transformation. The masking filter retains only the values ​​of the central part. In this embodiment, masking is performed to make the transformed feature with the lost values ​​at the boundary equivalent to the feature that has not been transformed at the boundary. Note that the post-processing is not limited to normalization or masking, and may be a trainable neural network such as a conditional generative network.

[0077] (Technical effect of this embodiment) By normalizing the points and masking the feature maps, the features (including transformed and untransformed features) become more suitable for comparison, which leads to more accurate angle estimation.

[0078] Embodiment 6. Fig. 12 is a block diagram showing an example of the configuration of an angle estimation device according to the sixth embodiment. The angle estimation device 201 shown in Fig. 12 includes an angle estimator 61. The angle estimator 61 is a device that has been trained by the training device 100.

[0079] The training apparatus 100 corresponds to any one of the training apparatuses 101 to 105 of the first to fifth embodiments. The angle estimator 61 corresponds to, for example, the first angle estimator 121 or the second angle estimator 122 extracted from any one of the training apparatuses 101 to 105 of the first to fifth embodiments.

[0080] Therefore, the angle estimator 61 can be trained by the training device 100 as described in the first to fifth embodiments.

[0081] The angle estimator 61 of this embodiment can correctly estimate the angle information of the input image.

[0082] Each component in the above embodiment can be configured with one piece of hardware, but can also be configured with one piece of software. Also, each component can be configured with multiple pieces of hardware, or can be configured with multiple pieces of software. Also, some of the components can be configured with hardware, and the other parts can be configured with software.

[0083] Each function (each process) in the above-described embodiments can be realized by a computer having a processor such as a CPU (Central Processing Unit), a memory, etc. For example, a program for implementing the method (process) in the above-described embodiments may be stored in a storage device (storage medium), and each function may be realized by executing the program stored in the storage device by a CPU.

[0084] 13 is a block diagram showing an example of a computer having a CPU. The computer is implemented in a training device and a classification device. The CPU 1000 executes processing according to a program stored in a storage device 1001 to realize each function in the above-mentioned embodiment. That is, by executing a program stored in the memory device 1001, the computer can realize the functions of the first feature extractor 111, the second feature extractor 112, the first angle estimator 121, the second angle estimator 122, the angle difference calculation unit 130, the first angle difference calculation unit 131, the second angle difference calculation unit 132, the rigid body transformation unit 140, the first rigid body transformation unit 141, the second rigid body transformation unit 142, the matching loss calculation units 150, 151, 152, the model update units 160, 161, the decoder 170, the first image pre-processing unit 181, the second image pre-processing unit 182, the first image post-processing unit 191, and the second image post-processing unit 192 in the training apparatus shown in Figures 1, 4, 6, 8, and 10.

[0085] The computer can realize the functions of the angle estimator 61 in the angle estimation device shown in FIG.

[0086] The storage device 1001 is, for example, a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage media. Specific examples of the non-transitory computer readable medium include a magnetic recording medium (for example, a hard disk), a magneto-optical recording medium (for example, a magneto-optical disk), a CD-ROM (Compact Disc-Read Only Memory), a CD-R (Compact Disc-Recordable), a CD-R / W (Compact Disc-ReWritable), and a semiconductor memory (for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), and a flash ROM).

[0087] The program may also be stored in various types of temporary computer-readable media, to which the program is supplied, for example, via a wired or wireless communication path, or via an electrical signal, an optical signal, or an electromagnetic wave.

[0088] The memory 1002 is realized by, for example, a random access memory (RAM), and is a storage means for temporarily storing data when the CPU 1000 executes processing. A configuration is also conceivable in which a program held in the storage device 1001 or a temporary computer-readable medium is transferred to the memory 1002, and the CPU 1000 executes processing based on the program in the memory 1002.

[0089] FIG. 14 is a block diagram showing the main parts of a training device. The training device 10 shown in FIG. 14 includes one or more feature extraction means 11 (realized by a first feature extractor 111 and a second feature extractor 112 in the embodiment), one or more angle estimation means 12 (realized by a first angle estimator 121 and a second angle estimator 122 in the embodiment) that estimates an angle from an input image, angle difference calculation means 13 that calculates an angle difference (realized by an angle difference calculation unit 130 in the embodiment), angle difference calculation means 13 that calculates the difference between the angles estimated by the one or more angle estimation means (realized by an angle difference calculation unit 130, or a first angle difference calculation unit 131 and a second angle difference calculation unit 132 in the embodiment), and rigid body transformation means 14 that transforms the features of the input image in accordance with the difference (realized by a rigid body transformation unit 140 or a first rigid body transformation unit 141 in the embodiment). The apparatus includes a first rigid body transformation unit 141 and a second rigid body transformation unit 142, a matching loss calculation unit 15 (in an embodiment, realized by a matching loss calculation unit 150, a matching loss calculation unit 151, or a matching loss calculation unit 152) that calculates a matching loss between a non-transformed feature (a feature not transformed by the rigid body transformation means 14) extracted by one or more feature extraction means 11 and a feature transformed by the rigid body transformation means 14, and an update means 16 (in an embodiment, realized by a model update unit 160 or a model update unit 161) that updates at least one or more angle estimation means 12 with reference to the matching loss, and the rigid body transformation means 14 transforms the feature so that it appears as if it was extracted from an image with the same angle as the image from which the non-transformed feature was extracted.

[0090] Fig. 15 is a block diagram showing the main parts of an angle estimation device. An angle estimation 20 shown in Fig. 15 includes an angle estimation means 21 (realized by an angle estimator 61 in the embodiment) that estimates an angle from an input image, and the angle estimation means 21 is trained together with one or more feature extraction means and rigid body transformation means so that an extracted image feature after being transformed according to the estimated angle appears as if it was extracted from an image at an angle different from that of the original input image.

[0091] In each of the above embodiments, the image is typically a SAR image. However, the image is not limited to a SAR image. As an example, the image can also be an optical image, such as an image taken by a smartphone.

[0092] Since the trained angle estimation device of the above embodiment can correctly estimate the angle information of the image, the trained angle estimation device can be incorporated into other image processing systems to provide angle information and improve the performance of the entire system. For example, when the angle estimation device of the above embodiment provides an estimated head pose of a human face image to a face recognition system, the recognition accuracy of the system is improved because the system has special knowledge about the human head pose.

[0093] Some or all of the above embodiments may be described as follows, but are not limited to the following:

[0094] (Appendix 1) One or more feature extraction means for extracting features from an input image; one or more angle estimators for estimating angles from an input image; An angle difference calculation means for calculating a difference between the angles estimated by the one or more angle estimation means; a rigid transformation means for transforming the features of the input image in accordance with the difference; a matching loss calculation means for calculating a matching loss between the non-transformed features extracted by the one or more feature extraction means and the features transformed by the rigid body transformation means; updating means for updating at least one or more angle estimators by referring to the matching loss; The rigid transformation means transforms the features so that they appear as if they were extracted from an image of the same angle as the image from which the untransformed features were extracted. Training equipment.

[0095] (Appendix 2) The update means also updates the one or more feature extraction means. 2. A training device as described in claim 1.

[0096] (Appendix 3) The apparatus further includes one or more preprocessing means for performing a predetermined preprocessing on the input image and supplying the preprocessed image to the one or more feature extraction means and the one or more angle estimation means. 10. The training device of claim 1 or 2.

[0097] (Appendix 4) The apparatus further includes one or more post-processing means for performing predetermined post-processing on the features extracted by the one or more feature extraction means and the features output by the rigid body transformation means. 10. The training device of claim 1 or 2.

[0098] (Appendix 5) One or more feature extraction means for extracting features from an input image; one or more angle estimators for estimating angles from an input image; one or more angle difference calculation means for calculating a difference between the angle estimated by the one or more angle estimation means and a reference view angle; one or more rigid body transformation means for transforming the non-transformed features extracted by the one or more feature extraction means in accordance with the difference; a matching loss calculation means for calculating a matching loss between features transformed by the one or more rigid body transformation means; updating means for updating at least the one or more angle estimation means and the one or more feature extraction means by referring to the matching loss; The rigid transformation means transforms the features so that they appear as if they were extracted from an image of the same angle as the image from which the untransformed features were extracted. Training equipment.

[0099] (Appendix 6) A feature extraction means for extracting features from an input image; one or more angle estimators for estimating angles from an input image; An angle difference calculation means for calculating a difference between the angles estimated by the one or more angle estimation means; a rigid transformation means for transforming the features of the input image in accordance with the difference; an image reconstruction means for reconstructing an image using the features extracted by the feature extraction means and an image using the features rigidly transformed by the rigid transformation means; a matching loss calculation means for calculating a matching loss between an image reconstructed from the transformed features and an input image having the same angle as the angle at which the features are transformed, and between an image reconstructed from the features without rigid body transformation and the input image from which the features are extracted; an update means for updating at least one of the angle estimation means, the feature extraction means and the image reconstruction means by referring to the matching loss; The rigid transformation means transforms the feature so that it appears as if it was extracted from an image at the same angle as the image from which the feature was extracted. Training equipment.

[0100] (Appendix 7) The matching loss calculation means repeatedly calculates the matching loss until the matching loss converges. 7. A training device according to any one of claims 1 to 6.

[0101] (Appendix 8) An angle estimation means for estimating an angle from an input image is provided, The angle estimation means, together with one or more feature extraction means and rigid body transformation means, is trained to make the image features appear as if they were extracted from an image at a different angle than the original input image after being transformed according to the estimated angle. Angle estimation device.

[0102] (Appendix 9) A training method for training a device having one or more angle estimation means, comprising: Extract features from the input image, estimating angles from the input images by one or more angle estimators; Calculate the difference in the estimated angles, The extracted features of the input image are rigidly transformed according to the difference; Calculate the matching loss between the untransformed features and the rigidly transformed features; updating at least one or more angle estimators with reference to the matching loss; The transformed features are transformed so that they appear as if they were extracted from an image with the same angle as the image from which the untransformed features were extracted. Training methods.

[0103] (Appendix 10) A device having one or more feature extraction means is trained; The one or more feature extraction means are updated with reference to the matching loss. 10. The training method according to claim 9.

[0104] (Appendix 11) Estimating angles from an input image using an angle estimation device that has been trained to, together with one or more feature extraction means and rigid transformation means, make the image features appear as if they were extracted from an image at a different angle than the original input image after being transformed according to the estimated angle. Angle estimation method.

[0105] (Appendix 12) A computer-readable information recording medium storing a training program for training a device having one or more angle estimation means, the training program being configured to: Extract features from the input image, causing one or more angle estimators to estimate angles from the input images; Calculate the difference in the estimated angles, The extracted features of the input image are rigidly transformed according to the difference; Calculate the matching loss between the untransformed features and the rigidly transformed features. updating at least one or more angle estimators with reference to the matching loss; The transformed features are transformed so that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted.

[0106] (Appendix 13) The training program includes a computer capable of implementing one or more feature extraction means for extracting features from an input image, The one or more feature extraction techniques are updated with reference to the matching loss. 13. The computer-readable storage medium of claim 12.

[0107] (Appendix 14) A computer-readable information recording medium having an angle estimation program recorded thereon, the angle estimation program being configured to: An angle is estimated from an input image using an angle estimator that has been trained in conjunction with one or more feature extraction means and rigid transformation means such that image features, after being transformed according to the estimated angle, appear as if they were extracted from an image at a different angle than the original input image.

[0108] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0109] 10,101~105 Training equipment 11 Feature extraction methods 12 Angle estimation means 13 Angle difference calculation means 14 Rigid body transformation methods 15 Matching loss calculation method 16 Update method 20,201 Angle estimation device 21 Angle estimation means 61 Angle Estimator 100 Training Equipment 111 First feature extractor 112 Second Feature Extractor 121 1st angle estimator 122 Second angle estimator 130 Angle Difference Calculation Section 131 First angle difference calculation unit 132 Second angle difference calculation unit 140 Rigid body transformation section 141 First rigid body transformation part 142 Second rigid body transformation part 150,151,152 Matching loss calculation section 160,161 Model Update Section 170 Decoder 181 First Image Preprocessor 182 Second Image Preprocessor 191 1st image post-processor 192 Second Image Post-Processor

Claims

1. one or more feature extraction means for extracting features from an input image; one or more angle estimators for estimating angles from an input image; An angle difference calculation means for calculating a difference between the angles estimated by the one or more angle estimation means; a rigid transformation means for transforming the features of the input image in accordance with the difference; a matching loss calculation means for calculating a matching loss between the non-transformed features extracted by the one or more feature extraction means and the features transformed by the rigid body transformation means; updating means for updating at least one or more angle estimators by referring to the matching loss; The rigid transformation means transforms the features so that they appear as if they were extracted from an image of the same angle as the image from which the untransformed features were extracted. Training equipment.

2. The updating means also updates the one or more feature extraction means.

2. The training device of claim 1.

3. The one or more preprocessing means further comprises: one or more preprocessing means for performing a predetermined preprocessing on the input image and supplying the preprocessed image to the one or more feature extraction means and the one or more angle estimation means. A training device according to claim 1 or 2.

4. The method further includes one or more post-processing means for performing predetermined post-processing on the features extracted by the one or more feature extraction means and the features output by the rigid body transformation means. A training device according to claim 1 or 2.

5. one or more feature extraction means for extracting features from an input image; one or more angle estimators for estimating angles from an input image; one or more angle difference calculation means for calculating a difference between the angle estimated by the one or more angle estimation means and a reference view angle; one or more rigid body transformation means for transforming the non-transformed features extracted by the one or more feature extraction means in accordance with the difference; a matching loss calculation means for calculating a matching loss between features transformed by the one or more rigid body transformation means; updating means for updating at least the one or more angle estimating means and the one or more feature extracting means by referring to the matching loss; The rigid transformation means transforms the features so that they appear as if they were extracted from an image of the same angle as the image from which the untransformed features were extracted. Training equipment.

6. A feature extraction means for extracting features from an input image; one or more angle estimators for estimating angles from an input image; An angle difference calculation means for calculating a difference between the angles estimated by the one or more angle estimation means; a rigid transformation means for transforming the features of the input image in accordance with the difference; an image reconstruction means for reconstructing an image using the features extracted by the feature extraction means and an image using the features rigidly transformed by the rigid transformation means; a matching loss calculation means for calculating a matching loss between an image reconstructed from the transformed features and an input image having the same angle as the angle at which the features are transformed, and between an image reconstructed from the features without rigid body transformation and the input image from which the features are extracted; an update means for updating at least one of the angle estimation means, the feature extraction means and the image reconstruction means by referring to the matching loss; The rigid transformation means transforms the feature so that it appears as if it was extracted from an image at the same angle as the image from which the feature was extracted. Training equipment.

7. The matching loss calculation means repeatedly calculates the matching loss until the matching loss converges. A training device according to any one of claims 1 to 6.

8. An angle estimation means for estimating an angle from an input image, The angle estimation means, together with one or more feature extraction means and rigid body transformation means, is trained to make the image features appear as if they were extracted from an image at a different angle than the original input image after being transformed according to the estimated angle. Angle estimation device.

9. 1. A method of training a device having one or more angle estimators, comprising: Extract features from the input image, estimating angles from the input images by one or more angle estimators; Calculate the difference in the estimated angles, The extracted features of the input image are rigidly transformed according to the difference; Calculate the matching loss between the untransformed features and the rigidly transformed features; updating at least one angle estimator with reference to the matching loss; The transformed features are transformed so that they appear as if they were extracted from an image with the same angle as the image from which the untransformed features were extracted. Training methods.

10. training a device having one or more feature extractors; The one or more feature extraction means are updated with reference to the matching loss. The training method according to claim 9.

11. Estimating angles from an input image using an angle estimator that has been trained in conjunction with one or more feature extraction means and rigid transformation means such that image features, after being transformed according to the estimated angles, appear as if they were extracted from an image at a different angle than the original input image. Angle estimation method.

12. A computer-readable information recording medium storing a training program for training a device having one or more angle estimation means, the training program being configured to cause a computer to: Extract features from the input image, causing one or more angle estimators to estimate angles from the input images; Calculate the difference in the estimated angles, The extracted features of the input image are rigidly transformed according to the difference; Calculate the matching loss between the untransformed features and the rigidly transformed features. updating at least one angle estimator with reference to the matching loss; The transformed features are transformed so that they appear as if they were extracted from an image at the same angle as the image from which the untransformed features were extracted.

13. The training program includes: The one or more feature extraction techniques are updated with reference to the matching loss.

13. The computer readable storage medium of claim 12.

14. A computer-readable information recording medium having an angle estimation program recorded thereon, the angle estimation program being configured to: An angle is estimated from an input image using an angle estimator that has been trained in conjunction with one or more feature extraction means and rigid transformation means such that image features, after being transformed according to the estimated angle, appear as if they were extracted from an image at a different angle than the original input image.

Images