Training equipment

Abstract

The feature extraction unit extracts source domain structural features from the input source domain image data, and extracts target domain structural features from the input target domain image data. The rigid transformation unit generates transformed structural features by performing rigid transformation on the structural features with reference to the transformation parameters. The relighting unit generates new view features with reference to the transformed structural features and the transformation parameters so as to approximate the structural features extracted in the view indicated by the transformation parameters. The class prediction unit performs source domain class prediction from the source domain structural features and the features of the new view of the source domain, and predicts a target domain class prediction value from the target domain structural features and the features of the new view of the target domain. The update unit updates at least one of the feature extraction unit, the relighting unit, and the class prediction unit.

Description

[Technical Field]

[0001] The present invention relates to a training device. to Regarding. [Background technology]

[0002] Neural networks require a large amount of labeled data to learn a huge number of parameters. Collecting a large amount of labeled data is costly and time-consuming. To solve this problem, it is possible to transfer knowledge from another domain to a new target domain (see, for example, Non-Patent Document 1).

[0003] Taking a classifier as an example, the "target domain" is the domain that the classifier targets. In this specification, the target domain is, for example, a set of actual Synthetic Aperture Radar (SAR) images. Another domain is called the "source domain." For example, the source domain is a set of simulated images automatically generated by a simulator based on the SAR imaging mechanism. However, source domain images can also be obtained in other ways. For example, the source domain images could be a set of actual SAR images acquired several years ago. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Xiang Xu et al., "d-SNE: Domain Adaptation using Stochastic Neighborhood Embedding", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] Using d-SNE, knowledge from the source domain can be transferred to the target domain. Take for example a classifier that classifies objects into class 1 or class 2. With d-SNE, objects in the target domain image can be classified based on the features of objects of the same category in the source domain image.

[0006] To safely transfer knowledge from a source domain to a target domain, two challenges must be overcome: domain gap and intra-class variance. First, different domains typically have different characteristics, such as image resolution, due to differences in data collection conditions. Such differences in characteristics cause domain gaps. Also, even within the same domain, images of objects collected from the same category may look very different due to other factors, such as different shooting angles and lighting conditions. This causes intra-class variance. Either of these unresolved challenges can cause a classifier to be unable to classify images in a new target domain with unaddressed target domain factors during the testing phase or beyond.

[0007] Because d-SNE does not consider intra-class variance when reducing the domain gap, it cannot simultaneously address both of these issues. As an example, consider a domain gap due to differences in image resolution and intra-class variance due to differences in image capture angles. Neural network-based classifiers can reduce the domain gap by training using both source and target domain images. However, if only a limited number of target domain images are available during the training phase, the domain gap cannot be properly minimized for factors that are present in the source domain images but not in the target domain training images. For example, during the training phase, the target domain may contain only images of class 1 objects and a small number of images of class 2 objects captured at one or two angles, while the source domain may contain images of class 1 objects and many images of class 2 objects captured at various angles. In this case, the domain gap between images captured at angles that are present in the source domain but not in the target domain cannot be properly reduced. During the testing phase, images of target domain objects at uncovered angles, which are not covered by the limited number of target domain training images, will not be correctly classified. This is because the domain gap at those angles is not minimized, and knowledge cannot be safely transferred from the source domain to the target domain at those angles. When minimizing the domain gap, d-SNE does not consider the intra-class variance from images taken at angles other than the one or two angles used in the training images of the target domain.

[0008] As a result, it becomes difficult for the classifier to determine which class a new image should be classified into, given factors in the target domain that have not yet been addressed. An example of a factor is the angle at which the image was taken. Other factors, such as lighting conditions, may also contribute to intra-class variance. When differences in factors exist, images from the same category taken at different angles will extract different features. The simplest definition of intra-class variance is the squared deviation of features from the same category from the population mean. Intra-class variance can also be expressed as a more complex variance between features extracted from images in the same category.

[0009] A domain gap refers to the difference in data distribution or characteristics between Domain A (in this example, the target domain) and Domain B (in this example, the source domain). For example, each collected dataset has different characteristics. An example of a characteristic is image resolution. There can also be other characteristics that may contribute to a domain gap, such as different image sensors or different object backgrounds.

[0010] When there is a difference in characteristics, the average features extracted from images belonging to domain A will differ from the average features extracted from images belonging to domain B. A simple definition of domain gap can be expressed as the distance between the two mean values. Alternatively, domain gap can be expressed as a more complex distance between data distributions of different domains. More specifically, domain gap can also be expressed as the maximum average discrepancy.

[0011] The object of the present invention is to provide a training device that enables a classifier to correctly classify a target domain in which new factors exist, even if the new factors are not seen in the learning data of the target domain by the classifier. of The purpose is to provide. [Means for solving the problem]

[0012] A training device according to one embodiment includes one or more feature extraction means for extracting source domain structural features from input image data of a source domain and extracting target domain structural features from input image data of a target domain; rigid body transformation means for generating transformed structural features by rigid body transforming the structural features with reference to transformation parameters; and one or more means for generating features of a new view with reference to the transformed structural features and the transformation parameters so as to approximate the structural features extracted from the input image data in a view indicated by the transformation parameters. New view feature generation one or more class prediction means for performing a class prediction for the source domain from the source domain structural features and the features of a new view of the source domain, and for performing a class prediction for the target domain from the structural features and the features of the new view of the target domain; one or more feature extraction means for calculating a merge loss from a source domain classification loss calculated with reference to a class prediction value for the source domain and a ground truth class label for the source domain, a target domain classification loss calculated with reference to a class prediction value for the target domain and a ground truth class label for the target domain, a transformation loss calculated with reference to the source domain structural features, the features of the new view of the source domain, the structural features of the target domain, and the features of the new view of the target domain, and a grouping loss calculated with reference to the source domain structural features, the features of the new view of the source domain, the structural features of the target domain, and the features of the new view of the target domain; and one or more New view feature generation and updating means for updating at least one of the one or more class prediction means. [Effects of the Invention]

[0018] According to the present invention, even if a classifier has factors that it has not seen in the training data of the target domain, it can correctly classify objects in the target domain in which the new factors exist. [Brief explanation of the drawings]

[0019] [Figure 1]1 is a block diagram showing an example of the configuration of a training device according to a first embodiment. [Figure 2] FIG. 10 is a block diagram illustrating an example of the configuration of a classification device according to a second embodiment. [Figure 3] FIG. 1 is an explanatory diagram for explaining structural features and structural transformations. [Figure 4] FIG. 1 is an explanatory diagram for explaining structural features. [Figure 5] FIG. 10 is a block diagram showing an example of the configuration of a training device according to a third embodiment. [Figure 6] FIG. 2 is an explanatory diagram illustrating input data of a target domain and a source domain. [Figure 7] FIG. 10 is an explanatory diagram for explaining cross-domain alignment. [Figure 8] 10 is a flowchart showing the operation of the training device of the third embodiment. [Figure 9] FIG. 10 is a block diagram illustrating an example of the configuration of a classification device according to a fourth embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the configuration of a training device according to a fifth embodiment. [Figure 11] 10 is a flowchart showing the operation of the training device of the fifth embodiment. [Figure 12] FIG. 13 is a block diagram showing an example of the configuration of a training device according to a sixth embodiment. [Figure 13] 10 is a flowchart showing the operation of the training device of the sixth embodiment. [Figure 14] FIG. 13 is a block diagram showing an example of the configuration of a training device according to a seventh embodiment. [Figure 15] 13 is a flowchart showing the operation of the training device of the seventh embodiment. [Figure 16] FIG. 1 is a block diagram illustrating an example of a computer having a CPU. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each embodiment described below, the image is assumed to be a SAR image. However, the image is not limited to a SAR image. As an example, the source domain and target domain images may be optical images, such as images taken by a smartphone.

[0021] Embodiment 1. FIG. 1 is a block diagram showing an example of the configuration of a training device according to the first embodiment.

[0022] The training device 10 shown in Fig. 1 includes a feature extraction unit 11, a rigid body transformation unit 12, a relighting unit 13, a class prediction unit 14, and an update unit 15. Although Fig. 1 shows one feature extraction unit 11, one rigid body transformation unit 12, one relighting unit 13, and one class prediction unit 14, a plurality of feature extraction units 11, rigid body transformation units 12, relighting units 13, and class prediction units 14 may be provided.

[0023] The feature extraction unit 11 extracts structural features from the input image data of the source domain, taking into account the structure of the source domain, and extracts structural features from the input image data of the target domain, taking into account the structure of the target domain.

[0024] The rigid transformation unit 12 transforms the structural features with reference to the transformation parameters to generate structurally transformed features. The transformed structural features are sometimes referred to as "transformed structural features."

[0025] The relighting unit 13 refers to the transformed structural features and the transformation parameters, and generates features of the new view indicated by the transformation parameters so as to approximate the structural features extracted from the input image data in the new view.

[0026] The class prediction unit 14 performs class prediction based on the structural features.

[0027] The update unit 15 updates at least one of the one or more feature extraction units 11, at least one of the one or more re-lighting units 13, and at least one of the one or more class prediction units 14. When updating, the update unit 15 refers to one or more of the following (1) to (4).

[0028] (1) The source domain classification loss calculated by referring to the predicted value of the source domain class and the source domain ground truth class label. (2) The target domain classification loss calculated by referring to the predicted value of the target domain class and the target domain ground truth class label. (3) A grouping loss calculated by referring to at least one feature from the structural features of the source domain, the novel view features of the source domain, the structural features of the target domain, and the corresponding class label of each feature associated with the novel view features of the target domain; (4) A transformation loss calculated by referring to at least one feature from the structural features of the source domain, the novel view features of the source domain, the structural features of the target domain, and the novel view features of the target domain.

[0029] Next, the "structural feature" and "structural feature conversion" will be described with reference to Figures 3 and 4. Figures 3 and 4 are explanatory diagrams for explaining the structural feature and the conversion of the structural feature.

[0030] In the example shown in Figures 3 and 4, the structural feature extraction function 110 extracts feature points Pa of the object 12A from an image 500 of view A. The feature points Pa form feature PA. The structural feature extraction function 110 extracts feature points Pb of the object 12B from an image 600 of view B. The feature points Pb form feature PB. As an example, the features PA and PB are features of a three-dimensional structure. The structural feature extraction function 110 is realized by the feature extraction unit 11. For example, the images 500 and 600 belong to the source domain. The objects 12A and 12B are projected onto a two-dimensional plane to form images 500 and 600, respectively. In the example shown in Figures 3 and 4, the object is a vehicle.

[0031] The feature converter, which is composed of the rigid body converter 12 and the relighting unit 13, performs a structural feature conversion. Specifically, the structural feature conversion converts the structural features in view A into structural features in view B.

[0032] Assume that feature points a1 and a2 are hidden and cannot be restored from image 500 of view A. In other words, feature point Pa other than feature points a1 and a2 can be restored. Also, assume that feature points b1 to b5 are hidden and cannot be restored from image 600 of view B. In other words, feature point Pb other than feature points b1 to b5 can be restored. In FIG. 3, only one feature point Pa is assigned a symbol, but all feature points Pa other than feature points a1 and a2 are restorable feature points. Also, only one feature point Pb is assigned a symbol, but all feature points Pb other than feature points b1 to b5 are restorable feature points.

[0033] The rigid transformation unit 12 rotates all feature points Pa clockwise along the z-axis so that the orientation of the object 12A faces the same direction as the object 12B. As an example, FIG. 4 illustrates an angle of 60°. The feature point Pa' is obtained by the rigid transformation. Hereinafter, this operation by the rigid transformation unit 12 may be referred to as a view change. As an example, the rigid transformation unit 12 rotates the feature point Pa so that the orientation of the object 12A faces the same direction as the object 12B. The rigid transformation unit 12 can move the object in any direction and rotate the object by any angle. Specifically, the rigid transformation unit 12 performs rigid transformation according to, for example, input transformation parameters.

[0034] The relighting unit 13 modifies the feature points Pa' based on the same transformation parameters received from the rigid transformation unit 12 to generate transformed feature points Pa''. The transformed feature points Pa'' restore the feature points a1 and a2 shown in FIG. 4 that could not be extracted from the image 500 in view A as shown in FIG. 3. Furthermore, the relighting unit 13 determines how the characteristics, such as intensity and color, of each feature point shown as an example in FIG. 4 should change after the view change. The transformed feature points Pa'' correspond to the new view features described above. The relighting unit is trained to perform such transformations by minimizing the matching loss. The matching loss is the difference between the transformed feature points Pa'' now in view B and the untransformed feature points Pb extracted from the image 600 in view B, as shown in FIG. 4.

[0035] By referencing multiple image pairs in the source domain, the rigid transformation and relighting units can suitably output new view features at the desired view by transforming features at any input view, and the new view features at the desired view are very similar to features extracted directly from the image at the desired view without transformation. Here, images 500 and 600 are images from the source domain. Structural knowledge is transferred from the source domain to the target domain by applying the structural feature extraction unit, rigid transformation unit, and relighting unit to the target domain image. Transforming the structural features of the target domain can create transformed feature points at the new view that were not available in the target domain.

[0036] [Technical effect of this embodiment] The data belonging to the source domain has a larger data size than the data belonging to the target domain. Furthermore, the data belonging to the source domain contains more labeled data than the data belonging to the target domain. "Labeled data" is, for example, data labeled with a "ground truth label." The labeled data can be used by a training device for both supervised learning and semi-supervised learning. In this embodiment, by transferring structural knowledge from an external dataset (i.e., the source domain) to the target domain, it is possible to train a classifier for general data in the target domain using not only a limited number of target domain data but also abundant data from the external dataset. As a result, classification accuracy is improved.

[0037] Embodiment 2. FIG. 2 is a block diagram illustrating an example of the configuration of a classification device according to the second embodiment.

[0038] The classification device 60 shown in FIG. 2 includes a feature extraction unit 61 and a class prediction unit 64.

[0039] The feature extraction unit 61 extracts structural features from input image data. The class prediction unit 64 predicts a class prediction value from the structural features. At least one of the feature extraction unit 61 and the class prediction unit 64 is trained by the training device 10 with reference to features of a new view obtained by transforming the structural features using the rigid body transformation unit 12 and the re-illumination unit 13.

[0040] [Technical effect of this embodiment] In this embodiment, the classification device 60 can perform a preferable classification process even when there are training images of a target domain with limited variations in shooting angles, for example.

[0041] Specific embodiments will be described below.

[0042] Embodiment 3. [Training equipment configuration] FIG. 5 is a block diagram showing an example of the configuration of a training device according to the third embodiment.

[0043] The training device 103 shown in FIG. 5 includes a first feature extraction unit 111, a second feature extraction unit 112, a first rigid body transformation unit 121, a second rigid body transformation unit 122, a first re-illumination unit 131, a second re-illumination unit 132, a first class prediction unit 141, a second class prediction unit 142, and an update unit 150.

[0044] Although unidirectional arrows are used in FIG. 5 and other figures, the unidirectional arrows are used to simply represent the flow of data and are not intended to exclude bidirectionality.

[0045] The first feature extraction unit 111 and the second feature extraction unit 112 each correspond to the feature extraction unit 11 shown in Fig. 1. The first rigid body transformation unit 121 and the second rigid body transformation unit 122 each correspond to the rigid body transformation unit 12 shown in Fig. 1. The first re-illumination unit 131 and the second re-illumination unit 132 each correspond to the re-illumination unit 13 shown in Fig. 1. The first class prediction unit 141 and the second class prediction unit 142 each correspond to the class prediction unit 14 shown in Fig. 1.

[0046] [Feature extraction section] The first feature extraction unit 111 and the second feature extraction unit 112 can be configured as a single section. The first rigid body transformation unit 121 and the second rigid body transformation unit 122 can be configured as a single section. The first re-illumination unit 131 and the second re-illumination unit 132 can be configured as a single section. The first class prediction unit 141 and the second class prediction unit 142 can be configured as a single section.

[0047] The update unit 150 includes a classification loss calculation unit 151 , a grouping unit 152 , a grouping loss calculation unit 153 , a conversion loss calculation unit 154 , a merge loss calculation unit 155 , and a model update unit 156 .

[0048] The first feature extraction unit 111 receives input image data (image data of the source domain) I S As an example, input image data I S may be an image having multiple regions. S may be a batch of images as depicted on the left side of Fig. 6. In the example shown on the left side of Fig. 6, the input image data I S is a set of four images (I S1 , I S2 , I S3 , I S4 ) The batch of images shown on the left in Figure 6 is a collection of simulated images. However, source domain images can be obtained in a variety of ways, for example, from existing databases. There are many existing datasets available online for scientific use, both free and for a fee.

[0049] Image I S1 and Image I S2 The relationship between Image I and Image II is as follows: S2 Here is the image I S1 For example, image I S2 Here is the image I S1 Image I is an image taken at a different angle from S1 Images containing the same subject as image I, or image I S1 It may also be an image containing another object in the same class category as image I. S1 and I S2 The images may be taken at the same time or at different times.

[0050] Similarly, image I S4 Here is the image I S3 As an example, image I S4 Here is the image I S3 Image I is an image taken at a different angle from S3 Images containing the same subject as image I, or image I S3It may also be an image containing another object in the same class category as image I. S3 and Image I S4 The images may be taken at the same time or at different times.

[0051] The first feature extraction unit 111 extracts the input source domain image data I S From the source domain, the features in the source domain (source domain structural features), i.e., the source domain feature quantity X S Specifically, the first feature extraction unit 111 extracts features of an object in an image belonging to the source domain as source domain features. The feature (feature) X extracted by the first feature extraction unit 111 is S is supplied to the first rigid body transform unit 121, the first class prediction unit 141, the grouping unit 152, and the transformation loss calculation unit 154.

[0052] For example, the first feature extraction unit 111 may be a convolutional neural network (CNN), a recurrent neural network (RNN), or any other neural network or feature extractor. However, the specific configuration of the first feature extraction unit 111 does not limit this embodiment or the following embodiments.

[0053] Source domain feature X S may be expressed in the form of a vector. S may be expressed as a vector as follows: However, the feature amount may be expressed in a format other than a vector.

[0054] X S = [x s1 , x s2 , x s3 , x s4 ] ...(Formula 1)

[0055] X S has four components, each of which is an input image (I S1 , I S2 , I S3 , IS4 ) Note that since features can be expressed as vectors, they are sometimes called feature vectors.

[0056] The second feature extraction unit 112 receives input image data I belonging to the target domain (image data I of the target domain). T As an example, input image data I T may be an image having multiple regions. T may be a batch of images as described on the right side of Fig. 6. In the example shown on the right side of Fig. 6, the input image data I T is a set of four images (I T1 , I T2 , I T3 , I T4 )

[0057] Image I T1 and Image I T2 The relationship between Image I and Image II is as follows: T2 Here is the image I T1 For example, image I T2 teeth, Image I T1 Image I is an image taken at a different angle from T1 Images containing the same subject as image I, or image I T1 It may also be an image containing another object in the same class category as image I. T1 and I T2 The images may be taken at the same time or at different times.

[0058] Similarly, image I T4 Here is the image I T3 As an example, image I T4 Here is the image I T3 Image I is an image taken at a different angle from T3 Images containing the same subject as image I, or image I T3 It may also be an image containing another object in the same class category as image I.T3 and Image I T4 The images may be taken at the same time or at different times.

[0059] The second feature extraction unit 112 extracts the input target domain image data I T From the target domain, the feature (target domain feature) X T Specifically, the second feature extraction unit 112 extracts features of the object in the image belonging to the target domain as target domain features. The feature (feature) X extracted by the second feature extraction unit 112 is T is supplied to the second rigid body transform unit 122, the second class prediction unit 142, the grouping unit 152, and the transformation loss calculation unit 154.

[0060] For example, the second feature extraction unit 112 may be a convolutional neural network (CNN), a recurrent neural network (RNN), or any other neural network or feature extractor. However, the specific configuration of the second feature extraction unit 112 does not limit this embodiment or the following embodiments.

[0061] Target domain feature X T may be expressed in the form of a vector. T may be expressed as a vector as follows: However, the feature amount may be expressed in a format other than a vector.

[0062] X T = [x T1 , x T2 , x T3 , x T4 ] ...(Formula 2)

[0063] X T has four components, each of which is an input image (I T1 , I T2 , I T3 , I T4) Note that since features can be expressed as vectors, they are sometimes called feature vectors.

[0064] [Rigid body transformation part] The first rigid body transformation unit 121 includes a structural transformation parameter Θ S is input, and the first feature extraction unit 111 extracts the source domain feature X S The first rigid body transformation unit 121 receives the source domain feature X S A rigid body transformation is performed on

[0065] For example, when the structural feature is expressed as a three-dimensional position (e.g., coordinate), the structural transformation parameter Θ S includes information on the direction of the rotation axis and the rotation angle. In this case, the first rigid transformation unit 121 performs the rigid transformation by executing the following calculation.

[0066]

number

[0067] In Equation 3, x, y, and z represent coordinates before transformation. x', y', and z' represent coordinates after transformation. θ represents the rotation angle. The matrix in Equation 3 rotates the object around the z-axis. However, in embodiments, it is possible to rotate the object around the x-axis or y-axis, as shown in Equation 3-2 or Equation 3-3.

[0068] As another example, when the structural features are expressed in voxels, the structural transformation parameter Θ S includes information on the direction of the rotation axis and the rotation angle. In this case, the first rigid transformation unit 121 performs rigid transformation by rotating each voxel by the rotation angle.

[0069] The rigid body transformation method is not limited to the above example, and various rigid body transformation methods can be applied. That is, the first rigid body transformation unit 121 can perform rigid body transformation using any rigid body transformation method. The first rigid body transformation unit 121 obtains the transformed structural features.

[0070] The second rigid body transformation unit 122 includes a structural transformation parameter Θ T is input, and the second feature extraction unit 112 extracts the target domain feature X T The second rigid body transformation unit 122 receives the target domain feature X T A rigid body transformation is performed on

[0071] Target domain feature X T When correcting, the second rigid body transformation unit 122 operates in the same manner as the first rigid body transformation unit 121 to perform rigid body transformation. Therefore, the second rigid body transformation unit 122 can obtain the transformed structural feature in the same manner as the first rigid body transformation unit 121.

[0072] [Re-lighting section] The first relighting unit 131 receives a structural transformation parameter Θ S is input to the first re-illumination unit 131. The first re-illumination unit 131 also receives the transformed structural features from the first rigid body transformation unit 121. The first re-illumination unit 131 corrects the transformed structural features from the first rigid body transformation unit 121.

[0073] The first relighting unit 131 calculates properties such as brightness, RGB color, and normal at each spatial position. If the structural feature is a 3D point cloud, every x, y, and z coordinate corresponds to one spatial position. If the structural feature is a collection of voxels, each voxel corresponds to one spatial position. The new value of the property for each point or voxel depends on the transformation parameters, the original value of the property, and the new position of the point or voxel. The new value corresponds to the structural feature of the transformed feature point (the transformed source domain structural feature). The original value corresponds to the structural feature of the feature point before transformation.

[0074] By the above operation, the first relighting unit 131 can obtain structural features based on structural features in one view as if they were extracted from an image from another view.

[0075] For example, the first relighting unit 131 may be a convolutional neural network (CNN), a recurrent neural network (RNN), or any other neural network or feature extractor. However, the specific configuration of the first relighting unit 131 does not limit this embodiment or the following embodiments.

[0076] The second relighting unit 132 receives the structural transformation parameter Θ T is input to the second re-illumination unit 132. The second re-illumination unit 132 also receives the transformed structural features from the second rigid body transformation unit 122. The second re-illumination unit 132 corrects the transformed structural features from the second rigid body transformation unit 122.

[0077] The second relighting unit 132 calculates properties such as brightness, RGB color, and normal at each spatial location. If the structural feature is a 3D point cloud, every x, y, and z coordinate corresponds to one spatial location. If the structural feature is a collection of voxels, each voxel corresponds to one spatial location. The new value of the property for each point or voxel depends on the transformation parameters, the original value of the property, and the new location of the point or voxel. The new value corresponds to the structural feature of the transformed feature point (the transformed target domain structural feature). The original value corresponds to the structural feature of the feature point before transformation.

[0078] By the above operation, the second relighting unit 132 can obtain structural features based on structural features in one view as if they were extracted from an image from another view.

[0079] For example, the second relighting unit 132 may be a convolutional neural network (CNN), a recurrent neural network (RNN), or any other neural network or feature extractor, but the specific configuration of the second relighting unit 132 does not limit this embodiment or the following embodiments.

[0080] [Class Prediction Section] The first class prediction unit 141 uses the source domain feature X extracted by the first feature extraction unit 111. S and the transformed feature (transformed source domain feature) X generated by the first re-illumination unit 131. ’S From this, the source domain class prediction value is predicted.

[0081] Specifically, the first class prediction unit 141 calculates the source domain feature X S From the source domain class prediction value (class probability) P S and predict the transformed source domain feature X' S Transformed source domain class prediction value (class probability) CP S Predict.

[0082] For example, the first class predictor 141 may be a convolutional neural network (CNN), a recurrent neural network (RNN), or any other neural network or feature extractor. However, the specific configuration of the first class predictor 141 does not limit this embodiment or the following embodiments.

[0083] Transformed source domain class prediction value CP S and the source domain class prediction value P S is supplied to the classification loss calculation unit 151.

[0084] For example, the first class predictor 141 may predict the source domain feature vector X S Each component of is compared with a threshold to obtain the source domain class prediction value P S and the transformed source domain feature vector X' S Each component of is compared with the same or different threshold to obtain the transformed source domain class prediction value CP S Determine.

[0085] As a specific example, the source domain feature vector X S and the transformed source domain feature vector X' STherefore, the first class prediction unit 141 calculates the source domain class prediction value P S and the source domain class predictor CP S may be output as follows:

[0086] P S = [0, 0, 1, 1] (Equation 4) CP S = [0, 0, 1, 1] (Equation 5)

[0087] P S has four components, which are the source domain feature vector X S Similarly, CP S has four components, which are the transformed source domain feature vector X' S The predicted values ​​correspond to the respective components of the vector. Since the predicted values ​​can be expressed as vectors, the predicted values ​​are sometimes called predicted vectors.

[0088] The second class prediction unit 142 uses the target domain feature X extracted by the second feature extraction unit 112. T and the transformed feature (transformed target domain feature) X generated by the second re-illumination unit 132. ’T From this, the target domain class prediction value is predicted.

[0089] Specifically, the second class prediction unit 142 calculates the target domain feature X T to the target domain class prediction value (class probability) P T and predict the transformed target domain feature X' T Transformed target domain class prediction value (class probability) CP T Predict.

[0090] For example, the second class predictor 142 may be a convolutional neural network (CNN), a recurrent neural network (RNN), or any other neural network or feature extractor. However, the specific configuration of the second class predictor 142 does not limit this embodiment or the following embodiments.

[0091] Target domain class prediction value CP after transformation T and the target domain class prediction value P T is supplied to the classification loss calculation unit 151.

[0092] For example, the second class predictor 142 may use the target domain feature vector X T Each component of is compared with a threshold to obtain the target domain class prediction value P T and the transformed target domain feature vector X' T Each component of is compared with a threshold or another threshold to obtain the transformed target domain class prediction value CP T Determine.

[0093] As a specific example, the target domain feature vector X T and the transformed target domain feature vector X' T Therefore, the second class prediction unit 142 calculates the target domain class prediction value P T and the target domain class predictor CP T may be output as follows:

[0094] P T = [0, 0, 1, 0] (Equation 6) CP T = [1, 0, 1, 0] (Equation 7)

[0095] P T has four components, which are the target domain feature vector X T Similarly, CP T has four components,

[0096] [Classification loss calculation part] The classification loss calculation unit 151 calculates the source domain class prediction value P S and the transformed predicted value CP S The source domain class prediction value and the source domain class label data Y S Calculate the source domain classification loss (Loss_classification_S) by referring to

[0097] Specifically, the classification loss calculation unit 151 calculates the source domain class prediction value P S , the transformed feature CP S , and the source domain class label data Y S For example, the classification loss calculation unit 151 calculates the source domain classification loss by referring to P S and Y S The degree of discrepancy between S and Y S Calculate the classification loss for the source domain according to the degree of discrepancy between

[0098] As a specific example, P S is given by Equation 4, and CP S is given by Equation 5, and Y Sが Consider the case given by the following equation 8.

[0099] Y S = [0, 0, 1, 1] (Equation 8)

[0100] The classification loss calculation unit 151 calculates P S All components of Y S and match the corresponding components of CP S All components of Y S Since it matches each corresponding component of, we calculate the classification loss of the source domain as follows:

[0101] Loss_classification_S = 0 (Equation 9)

[0102] In addition, the classification loss calculation unit 151 calculates the target domain class prediction value P T and the transformed predicted value CP T The target domain class prediction value and the target domain class label data Y T Calculate the target domain classification loss (Loss_classification_T) by referring to

[0103] Specifically, the classification loss calculation unit 151 calculates the target domain class prediction value P S , the transformed feature CP S , and the target domain class label data Y S Calculate the target domain classification loss by referring to

[0104] As a specific example, P T is given by Equation 6, and CP T is given by Equation 7, and Y Tが Consider the case given by the following equation 10.

[0105] Y T = [0, 0, 1, 1] (Equation 10)

[0106] The classification loss calculation unit 151 calculates P T and the fourth component of YT The fourth component of CP T The first and fourth components of Y T Since the corresponding components of do not match, the classification loss for the target region is calculated as follows:

[0107] Loss_classification_T = 3 (Equation 11)

[0108] [Grouping section] The grouping unit 152 groups the source domain feature X S , the transformed source domain feature X' S , target domain feature X T , the transformed target domain feature X' TFrom the dataset, generate and output class groups containing features that share the same class label.

[0109] Let the class groups be Gr0 and Gr1. Gr0 is a class group that shares the same feature amount and class label 0. Gr1 is a class group that shares the same feature amount and class label 1.

[0110] [Grouping loss calculation part] The grouping loss calculation unit 153 refers to the class groups generated by the grouping unit 152 and calculates a grouping loss (Loss_grouping).

[0111] For example, the grouping loss calculation unit 153 calculates the grouping loss based on intra-class metrics determined with reference to the feature amounts of the same class and inter-class metrics determined with reference to the feature amounts of different classes.

[0112] As a specific example, the grouping loss calculation unit 153 calculates the grouping loss using the following formula:

[0113]

number

[0114] This formula calculates the average difference between the maximum intra-class distance and the minimum inter-class distance, plus the margin across all class groups Gr. For each class group g, the maximum intra-class distance is calculated as the maximum distance between any two feature values ​​within group g, and the minimum inter-class distance is calculated as the minimum distance between any two feature values ​​where one feature value is from group g and the other feature value is from a different group. The margin indicates the minimum allowable difference between the maximum intra-class distance and the minimum inter-class distance in each class group. The average is calculated by first summing the distance differences plus the margin across all class groups, and then dividing that sum by the number of class groups.

[0115] Specifically, the calculation of the grouping loss (Loss_grouping) according to Equation 12 can be expressed as follows:

[0116] The grouping loss calculation unit 153 first finds, for each class group, a pair of feature values ​​that are the most distant from each other in the feature space. Such pairs may be called intra-class pairs. The maximum distance between the features of the intra-class pair for each class corresponds to the "maximum intra-class distance in the feature space" in Equation 12.

[0117] The grouping loss calculation unit 153 finds, for each class group in the feature space, a pair of features that belong to different classes and are closest to each other. Such pairs are sometimes called inter-class pairs. The minimum value of the distance between the features of an inter-class pair corresponds to the "minimum value of the inter-class distance in the feature space" in Equation 12.

[0118] Then, for each class group, the grouping loss calculation unit 153 subtracts the minimum inter-class distance in the feature space from the maximum intra-class distance in the feature space. Note that the grouping loss may be calculated by any method for calculating the distance or similarity between two features in a class group. For example, the grouping loss may be the L1 norm, the L2 norm, cosine similarity, or another index that requires learning.

[0119] The grouping loss calculation unit 153 may add a margin. Adding a margin means that the maximum value of the intra-class distance in the feature space is desired to be smaller by at least a certain value than the minimum value of the inter-class distance in the feature space (for example, if the margin=1, then for each class group, the maximum value of the intra-class distance in the feature space is set to be smaller by one unit or more than the minimum value of the inter-class distance in the feature space).

[0120] The grouping loss calculation unit 153 may perform the above calculation for each class group and then take the sum of all the class groups.

[0121] Next, the grouping loss calculation unit 153 divides the sum result by the number of all class groups, which is represented by g in Equation 12.

[0122] Note that, for example, grouping losses may be calculated separately for real features and transformed features. "Real" refers to a grouping loss calculated with reference to real features that do not pass through the first rigid body transformation unit 121, the first re-illumination unit 131, the second rigid body transformation unit 122, and the second re-illumination unit 132. "Transformed" refers to a grouping loss calculated from real features with reference to transformed features generated by the first rigid body transformation unit 121, the first re-illumination unit 131, the second rigid body transformation unit 122, and the second re-illumination unit 132.

[0123] In other words, the grouping loss calculation unit 53 calculates X S and X T The grouping loss calculation unit 53 may calculate the grouping loss of the actual feature based on the feature from the combination of X' S and X' T You can also compute a separate grouping loss for the transformed features based on features from the combination (including only the transformed features).

[0124] Alternatively, the group loss may be calculated after undesirable features are filtered out based on a specific condition, which may depend on the accuracy of the predictions provided by the first class predictor 141 and the second class predictor 142, or on the reliability of the predictions provided by the first class predictor 141 and the second class predictor 142.

[0125] [Conversion loss calculation section] The conversion loss calculation unit 154 calculates the source domain feature X S , the transformed source domain feature X' S , target domain feature X T, and the transformed target domain feature X' T Calculate the conversion loss (Loss_conversion) by referring to

[0126] The conversion loss calculation unit 154 calculates the source domain feature X S and the corresponding transformed source domain feature X' S and the target domain feature X T and the corresponding transformed target domain feature X' T Calculate the conversion loss based on the difference between

[0127] For example, the conversion loss calculation unit 154 calculates the conversion loss based on the following formula:

[0128]

number

[0129] In Equation 13, x S1 , x S2 , x S3 , x S4 are source domain features, respectively. x S1 and x S2' and are from the same perspective. x S2 and x S1' and are from the same perspective. x S3 and x S4' and are from the same perspective. x S4 and x S3' and are from the same perspective. x T1 , x T2 , x T3 , x T4 are the target domain features. x T1 and x T2' and are from the same perspective. x T2 and x T1' and are from the same perspective. x T3 and x T4' and are from the same perspective. x T4 and x T3'are from the same view. In Equation 13, the difference between the same view feature and the same domain feature is summed to calculate the conversion loss (Loss_conversion).

[0130] [Merge loss calculation part] The merge loss calculation unit 155 calculates the merge loss (Loss_merge) by referring to the source domain classification loss (Loss_classification_S), the target domain classification loss (Loss_classification_T), the grouping loss (Loss_grouping), and the conversion loss (Loss_conversion).

[0131] For example, merge loss calculation unit 155 calculates the merge loss as follows.

[0132] Loss_merge = αLoss_classification_S + βLoss_classification_T + γLoss_grouping + δLoss_conversion (Equation 14)

[0133] In Equation 14, α, β, γ, and δ are weighting coefficients. The weights may change depending on the progress of the training. For example, in the early training iterations, the weights of the classification loss are large, and then the weights decay as the number of training iterations increases. Alternatively, the weights may be weights that require training.

[0134] [Model Update Section] The model update unit 156 determines whether the merge loss has converged. If the merge loss has converged, the model update unit 156 outputs the model parameters to a storage medium. If the merge loss has not converged, the model update unit 156 refers to the merge loss calculated by the merge loss calculation unit 155 and updates the model parameters of the first feature extraction unit 111, the second feature extraction unit 112, the first re-illumination unit 131, the second re-illumination unit 132, the first class prediction unit 141, and the second class prediction unit 142.

[0135] For example, the model update unit 156 updates the model parameters so as to reduce the merge loss. As an example, the model update unit 156 updates the model parameters according to the gradient backpropagation method.

[0136] The model parameters updated by the model update unit 156 are supplied to the first feature extraction unit 111, the second feature extraction unit 112, the first relighting unit 131, the second relighting unit 132, the first class prediction unit 141, and the second class prediction unit 142.

[0137] [Technical effect of this embodiment] In this embodiment, the model update unit 156 updates the model parameters by referring to the grouping loss in addition to the source domain classification loss, the target domain classification loss, and the transformation loss. Therefore, in the third embodiment, the source domain features and the target domain features preferably overlap each other, while the features belonging to different classes are preferably separated for each class in the feature space.

[0138] 7 is an explanatory diagram schematically illustrating cross-domain registration achieved in this embodiment. In the training device, since the grouping loss is included in the merging loss, as learning progresses, the features of the source domain and the target domain preferably overlap with each other, while the features belonging to different classes are preferably separated by class in the feature space, as shown in FIG. 7. That is, in this embodiment, cross-domain registration in the feature space is appropriately achieved.

[0139] As a result, even when there is little labeled data in the target domain, the second feature extraction unit 112 and the second class prediction unit 142 are appropriately trained.

[0140] Furthermore, the first rigid body transformation unit 121 calculates the structural transformation parameter Θ S According to the source domain feature X SThen, the first re-illumination unit 131 corrects the transformed source domain feature X' based on the structural feature transformed by the first rigid body transformation unit 121. S The second rigid body transformation unit 122 generates the structural transformation parameter Θ T According to the target domain feature X T Then, the second re-illumination unit 132 corrects the transformed target domain feature X' based on the structural feature transformed by the second rigid body transformation unit 122. T Generate.

[0141] The first re-illumination unit 131 can obtain, based on the structural features in a certain view, structural features that appear to have been extracted from an image from another view indicated by the structural transformation parameters. The second re-illumination unit 132 can obtain, based on the structural features in a certain view, structural features that appear to have been extracted from an image from another view indicated by the structural transformation parameters.

[0142] Therefore, the first class prediction unit 141 and the second class prediction unit 142 can learn to perform appropriate class prediction for various shooting angles.

[0143] [Training equipment operation] Next, the operation of the training device 103 will be described with reference to the flowchart of FIG.

[0144] The training device 103 receives initial model parameters (step S100). The initial model parameters include initial model parameters of the first feature extraction unit 111, the second feature extraction unit 112, the first re-illumination unit 131, the second re-illumination unit 132, the first class prediction unit 141, and the second class prediction unit 142. The received initial model parameters are supplied to the first feature extraction unit 111, the second feature extraction unit 112, the first re-illumination unit 131, the second re-illumination unit 132, the first class prediction unit 141, and the second class prediction unit 142.

[0145] The training device 103 receives the input source domain data. That is, the training device 103 receives the source domain image data I S and image data I Sに Associated source domain class label data Y S (Step S101A).

[0146] The first rigid body transformation unit 121 calculates the source domain structural transformation parameter Θ S (Step S102A).

[0147] The first feature extraction unit 111 extracts image data I S From source domain feature X S is extracted (step S111).

[0148] The first rigid body transformation unit 121 calculates the structural transformation parameter Θ S Based on this, the source domain features (source domain structural features) X S (Step S121). The first re-illumination unit 131 performs rigid body transformation on the structure transformation parameter Θ T Based on this, the transformed structural features from the first rigid body transformation unit 121 are corrected (step S131).

[0149] The first class prediction unit 141 predicts the source domain feature X S From the source domain class prediction value (probability) P S is predicted (step S141).

[0150] The training device 103 receives input target domain data. That is, the training device 103 receives target domain image data I T and image data I T The class label data Y of the target domain associated with T (Step S101B).

[0151] The second rigid body transformation unit 122 calculates the target domain structural transformation parameter Θ T(Step S102B).

[0152] The second feature extraction unit 112 extracts the image data I of the target object. T to target domain feature X T is extracted (step S112).

[0153] The second rigid body transformation unit 122 calculates the structural transformation parameter Θ T Based on the target domain feature X S (Step S122). The first re-illumination unit 131 performs rigid body transformation on the structure transformation parameter Θ T Based on this, the transformed structural features from the second rigid body transformation unit 122 are corrected (step S132).

[0154] The second class prediction unit 142 predicts the target domain feature X T From the target domain class prediction value (probability) P T is predicted (step S142).

[0155] The classification loss calculation unit 151 calculates the transformed source domain feature CP S The source domain class prediction value P S and the source domain class label data Y S The classification loss calculation unit 151 calculates the source domain classification loss (Loss_classification_S) by referring to the target domain class prediction value P T and the transformed target domain feature CP T and the target domain class label data Y T and calculate the target domain classification loss (Loss_classification_T) (step S151).

[0156] The grouping unit 142 groups the source domain structural features (source domain features) X S , the transformed source domain feature X' S, target domain structural features (target domain features) X T , the transformed target domain feature X' T Then, class groups each containing features that share the same class label are generated and output (step S152).

[0157] The grouping loss calculation unit 153 calculates the grouping loss (Loss_grouping) by referring to the class groups generated by the grouping unit 152 (step S153). S , the transformed source domain feature X' S , target domain feature X T , and the transformed target domain features X' T The conversion loss is calculated by referring to (step S154).

[0158] The merge loss calculation unit 155 calculates the merge loss (Loss_merge) by referring to the source domain classification loss (Loss_classification_S), the target domain classification loss (Loss_classification_T), the grouping loss (Loss_grouping), and the conversion loss (Loss_conversion) (step S155). The merge loss calculation unit 155 calculates the merge loss using, for example, Equation 14.

[0159] The model update unit 156 determines whether the merge loss has converged (step S156). If the combined loss has converged (Yes in step S156), the process proceeds to step S158. If the combined loss has not converged (No in step S156), the process proceeds to step S157. Note that the model update unit 156 may, for example, compare the merge loss with a predetermined threshold to determine whether the merge loss has converged.

[0160] The model update unit 156 refers to the merge loss calculated by the merge loss calculation unit 155 and updates the model parameters of the first feature extraction unit 111, the second feature extraction unit 11b, the first re-illumination unit 131, the second re-illumination unit 132, the first class prediction unit 141, and the second class prediction unit 142 (step S157).

[0161] The model update unit 156 stores the model parameters of the first feature extraction unit 111, the second feature extraction unit 11b, the first re-illumination unit 131, the second re-illumination unit 132, the first class prediction unit 141, and the second class prediction unit 142 in a storage medium (not shown in FIG. 5).

[0162] The order of execution of steps S151, S152, and S153 and the order of execution of step S154 are arbitrary. They may be executed in any order. Furthermore, the order of execution of steps S151, S152, and S153 and the order of execution of step S154 may be executed simultaneously.

[0163] Embodiment 4. [Training equipment configuration] FIG. 9 is a block diagram illustrating an example of the configuration of a classification device according to the fourth embodiment.

[0164] 9 includes a feature extraction unit 61, a rigid body transformation unit 62, a relighting unit 63, and a class prediction unit 64. The feature extraction unit 61 and the class prediction unit 64 are the same as those in the second embodiment.

[0165] The rigid body transformation unit 62, like the second rigid body transformation unit 122 shown in FIG. 5, T Similar to the second relighting unit 132, the relighting unit 63 can obtain structural features based on structural features in one view as if they were extracted from an image from another view.

[0166] The rigid body transformation unit 62 operates in the same manner as the second rigid body transformation unit 122. That is, the rigid body transformation unit 62 transforms the target domain feature X TThe relighter 63 operates in a similar manner to the second relighter 132, i.e., it modifies the transformed structural features from the rigid transformer 62.

[0167] [Technical effect of this embodiment] In this embodiment, the classification device 70 can perform appropriate class prediction for input images captured at various angles, even when there are training images with limited variations in the captured angles, for example.

[0168] Embodiment 5. [Training equipment configuration] FIG. 10 is a block diagram showing an example of the configuration of a training apparatus according to the fifth embodiment. The training apparatus 104 shown in FIG. 10 has a configuration in which a domain alignment unit 211 and a domain alignment loss calculation unit 212 are added to the training apparatus 103 according to the third embodiment shown in FIG. 5. However, in this embodiment, the merge loss calculation unit 155 also merges the domain alignment loss. Various options are available for domain alignment (domain registration). The domain alignment unit 211 can be implemented to select a kernel for calculating the maximum mean disparity (MMD) between the source domain and the target domain, and the domain alignment loss calculation unit 212 calculates the MMD as the loss. As another example, the domain alignment unit 211 can be implemented as a domain discrimination unit, and the domain alignment loss calculation unit 212 can be implemented as a domain confusion loss calculation unit. An example of the latter is described below.

[0169] The domain matching unit 211 performs domain matching processing to match the target domain and the source domain. For example, the domain matching unit 211 may be a convolutional neural network (CNN), a recurrent neural network (RNN), other neural networks, or a feature extractor. However, the specific configuration of the domain matching unit 211 is not limited to this embodiment and the following embodiments.

[0170] For example, the domain matching unit 211 as a domain discrimination unit performs domain prediction to indicate whether the feature is from the source domain or the target domain. The domain matching unit 211 performs domain prediction to indicate whether the feature is from the source domain or the target domain. S and the target domain feature X extracted by the second feature extraction unit 112. T Then, the domain matching unit 211 receives the source domain feature X without referring to other information about which domain it belongs to. S and target domain feature X T For example, in the discrimination process, the domain matching unit 211 calculates the probability that each feature is from the source domain, calculates the probability that each feature is from the target domain, and assigns the domain label with the higher probability as the predicted domain label of the feature. The result of the discrimination process is then output. Note that the transformed features may also be involved in the discrimination process.

[0171] Source domain feature X S The classification results for S Let the target domain feature X T The classification results for T Let's say.

[0172] The domain matching loss calculation unit 212 calculates the domain matching loss according to the distance (e.g., MMD) between the source domain and the target domain. When the domain matching loss calculation unit 212 operates as a domain confusion loss calculation unit, the domain matching loss calculation unit 212 calculates the domain matching loss according to the DP S and source domain label data D S the degree of disagreement with the T and target domain label data D T The domain matching loss can also be calculated according to the degree of mismatch between the two. The degree of mismatch is calculated, for example, using a binary cross-entropy error function. The domain matching loss calculation unit 212 calculates the domain matching loss by, for example, summing the degrees of mismatch between the two.

[0173] In this embodiment, the merge loss calculation unit 155 calculates the domain matching loss as follows.

[0174] Loss_merge = αLoss_classification_S + βLoss_classification_T + γLoss_grouping +δLoss_conversion - τLoss_domain_alignment (Equation 15)

[0175] In Equation 15, Loss_domain_alignment indicates the domain alignment loss. τ is a weighting coefficient. Note that the sign before the domain alignment loss is negative. This means that the model update unit 156 updates the model parameters of the first feature extraction unit 111 and the second feature extraction unit 112 so that the extracted features may reduce the accuracy of the classification result by the domain alignment unit 211. In other words, the model update unit 156 updates the model parameters of the first feature extraction unit 111 and the second feature extraction unit 112 so that the extracted features may confuse the domain alignment unit 211.

[0176] During training, the training device 104 performs the following processes. First, the training device 104 trains the domain matching unit 211 so that the domain matching unit 211 can distinguish whether a feature is from the source domain or the target domain. Next, the training device 104 trains the first feature extraction unit 111 and the second feature extraction unit 112 so that they extract features that may confuse the trained domain matching unit 211. The training device 104 repeats the above processes.

[0177] It should be noted that in this embodiment, the domain gap between the source domain and the target domain is further minimized, i.e., structural knowledge can be transferred more accurately from the source domain to the target domain.

[0178] [Training equipment operation] Next, the operation of the training device 104 will be described with reference to the flowchart in Fig. 11. The processing of steps S100 to S154 and the processing of steps S156 to S158 are the same as the operation of the training device 104 of the third embodiment shown in Fig. 8.

[0179] In this embodiment, in step S211, the domain matching unit 211 executes the above-described domain matching process including the determination process. In step S212, the domain matching loss calculation unit 212 calculates the domain matching loss based on the result of the matching process by the domain matching unit 211.

[0180] In step S155B, the merge loss calculation unit 155 calculates a merge loss (Loss_merge) by referring to the source domain classification loss (Loss_classification_S), the target domain classification loss (Loss_classification_T), the grouping loss (Loss_grouping), the conversion loss (Loss_conversion), and the domain alignment loss (Loss_domain_alignment). The merge loss calculation unit 155 calculates the merge loss using, for example, Equation 15.

[0181] [Technical effect of this embodiment] By introducing a technique called "domain confusion," it is possible to reduce the domain gap between the source domain, the target domain, and the transformed target, which is the transformed target domain features generated from the target domain features.

[0182] Embodiment 6. [Training equipment configuration] Fig. 12 is a block diagram showing an example of the configuration of a training device according to the sixth embodiment. The training device 105 shown in Fig. 12 has a configuration in which an auxiliary task solver 311, an auxiliary task solver 312, an auxiliary loss calculation unit 321, and an auxiliary loss calculation unit 322 are added to the training device 103 according to the third embodiment shown in Fig. 5. However, in this embodiment, the merge loss calculation unit 155 also merges the auxiliary loss.

[0183] In machine learning, the quality of features extracted from images has a significant impact on the performance of the model, so it is desirable to be able to extract higher quality features by satisfying not only the final classification goal but also secondary goals.

[0184] In this embodiment, auxiliary tasks are introduced to satisfy secondary goals as well. An auxiliary task solver 311 and an auxiliary task solver 312 solve the auxiliary tasks. Various types of auxiliary tasks can be applied.

[0185] Option 1. In the first option, the auxiliary task is assumed to be an image restoration task, which is to restore the original image from features extracted from the image. The auxiliary loss calculation units 321, 322 can be realized as a decoding neural network that takes the features as input and outputs an image reconstructed from the features. They calculate the pixel-by-pixel intensity difference between the input image and the reconstructed image. When the loss is minimized, the reconstructed image is almost identical to the input image. This means that the features extracted from the image are a good compression of the original image, with only a small amount of information lost.

[0186] Specifically, as shown in Figure 12, the source domain structural feature X S is input from the first feature extraction unit 111 to the auxiliary task solver 311. The converted source domain structural feature X ’S is input from the first relighting unit 131 to the auxiliary task solver 311.

[0187] The auxiliary task solver 311 receives input image data (image data in the source domain) I S Refer to the source domain feature X S and the transformed source domain structural feature X ’S For example, the auxiliary task solver 311 generates a reconstructed image from the source domain feature x S1 and the transformed source domain features x from the first relighting unit 131. S2' The two reconstructed images generated by the auxiliary task solver 311 are based on the source domain features x S1 is very similar to the image from which feature x was extracted. S1 and x S2' And it's in the same view.

[0188] Auxiliary label data Y ’S is input to the auxiliary loss calculation unit 321. The auxiliary label data Y ’S is used to calculate the auxiliary loss. In this option, the auxiliary label data Y ’S is the input image. The auxiliary loss calculation unit 321 calculates the auxiliary label data Y ’S The difference between the source domain and the reconstructed image is calculated to obtain the source domain reconstruction loss (Loss_reconstruction_S) as an auxiliary loss.

[0189] As shown in Figure 12, the target domain structural feature X T is input from the second feature extraction unit 112 to the auxiliary task solver 312. The transformed target domain structural feature X ’T is input to the auxiliary task solver 312 from the second relighting unit 132.

[0190] The auxiliary task solver 312 receives input image data (image data in the target domain) I T Refer to the target domain feature X T and the transformed target domain structural feature X ’T For example, the auxiliary task solver 312 generates a reconstructed image from the target domain feature x T1and the transformed target domain features x from the second relighting unit 132. T2' The two reconstructed images generated by the auxiliary task solver 312 are the target domain features x T1 is very similar to the image from which feature x was extracted. T1 and x T2' And it's in the same view.

[0191] Auxiliary label data Y ’T is input to the auxiliary loss calculation unit 322. The auxiliary label data Y ’T is used to calculate the auxiliary loss. In this option, the auxiliary label data Y ’T is the input image. The auxiliary loss calculation unit 322 calculates the auxiliary label data Y ’T The difference between the target domain reconstruction loss and the reconstructed image is calculated to obtain the target domain reconstruction loss (Loss_reconstruction_T) as an auxiliary loss.

[0192] In this option, the merge loss calculation unit 155 calculates the auxiliary loss as follows: In Equation 16, η and ξ are weighting coefficients.

[0193] Loss_merge = αLoss_classification_S + βLoss_classification_T + γLoss_grouping +δLoss_conversion + ηLoss_reconstruction_S + ξLoss_reconstruction_T (Equation 16)

[0194] Option 2. In the second option, an angle estimation task is envisaged as the auxiliary task, which can estimate the angle of the image from the features extracted from the image. The auxiliary task solvers 321 and 322 can be implemented as angle estimation neural networks that take the features as input and output angle values ​​within [-π,π] ([-180°, 180°]).

[0195] Specifically, as shown in Figure 12, the source domain structural feature X S is input from the first feature extraction unit 111 to the auxiliary task solver 311. The converted source domain structural feature X ’S is input from the first relighting unit 131 to the auxiliary task solver 311.

[0196] The auxiliary task solver 311 calculates the source domain structure feature X S and the transformed source domain structural feature X ’S The source domain angle prediction value is predicted from the

[0197] Auxiliary label data Y ’S is input to the auxiliary loss calculation unit 321. The auxiliary label data Y ’S is used to calculate the auxiliary loss. In this option, the auxiliary label data Y ’S is the true angle of the input image. The auxiliary loss calculation unit 321 calculates the difference between the source domain angle prediction value and the true angle to obtain the source domain angle prediction loss (Loss_angle_prediction_S) as the auxiliary loss.

[0198] When the auxiliary loss (angle prediction loss) is minimized, the estimated angle is nearly identical to the true angle of the input image. This means that the features extracted from the image contain angle cues in addition to class cues of the input image. Having more informative features like this makes the classification task easier and improves classification accuracy, which is the ultimate goal.

[0199] As shown in Figure 12, the target domain structural feature X T is input from the second feature extraction unit 112 to the auxiliary task solver 312. The transformed target domain structural feature X ’T is input to the auxiliary task solver 312 from the second relighting unit 132.

[0200] The auxiliary task solver 312 calculates the target domain structural features X T and the transformed target domain structural feature X ’TThe target domain angle prediction value is predicted from the

[0201] Auxiliary label data Y ’T is input to the auxiliary loss calculation unit 322. The auxiliary label data Y ’T is used to calculate the auxiliary loss. In this option, the auxiliary label data Y ’T is the true angle of the input image. The auxiliary loss calculation unit 322 calculates the difference between the target domain angle prediction value and the true angle to obtain the target domain angle prediction loss (Loss_angle_prediction_T) as the auxiliary loss.

[0202] In this option, the merge loss calculation unit 155 calculates the auxiliary loss as follows: In Equation 17, η′ and ξ′ are weighting coefficients.

[0203] Loss_merge = αLoss_classification_S + βLoss_classification_T + γLoss_grouping +δLoss_conversion + η'Loss_angle_prediction_S + ξ'Loss_angle_prediction_T (Equation 17)

[0204] Option 3. In the third option, a conversion confusion task is envisaged as an auxiliary task, which minimizes the domain gap given the transformed and untransformed domains.

[0205] The concept of "transformation confusion" in this option is almost the same as "domain confusion" in the fifth embodiment. "Transformation confusion" is considered to be a case where the "transformation discrimination module" is very good at distinguishing between features from the transformed domain and features from the untransformed domain. Here, the transformed domain consists of features transformed from the source domain and features transformed from the target domain, and the untransformed domain consists of untransformed features from the source domain and features untransformed from the target domain. However, in this option, the "feature extraction module" is intentionally trained. The feature extraction module extracts features that are so mixed that even a powerful "transformation discrimination module" cannot distinguish between the transformed and untransformed domains. This minimizes the domain gap between the transformed and untransformed domains.

[0206] Specifically, as shown in Figure 12, the source domain structural feature X S is input from the first feature extraction unit 111 to the auxiliary task solver 311. The converted source domain structural feature X ’S is input from the first relighting unit 131 to the auxiliary task solver 311.

[0207] The auxiliary task solver 311 calculates the source domain structure feature X S Specifically, the auxiliary task solver 311 calculates the probability that each feature has been transformed by the first rigid body transformation unit 121 and the first relighting unit 131 and the probability that each feature has not been transformed, and uses the transformation label with the higher probability as the predicted value.

[0208] Auxiliary label data Y ’S is input to the auxiliary loss calculation unit 321. The auxiliary label data Y ’S is used to calculate the auxiliary loss. In this option, the auxiliary label data Y ’S For each feature, the correct transformed label data is either "transformed domain" or "transformed domain". The auxiliary loss calculation unit 321 calculates the auxiliary label data Y’S We obtain the auxiliary loss,conversion confusion loss (Loss_conversion_confusion_S), by calculating the correspondence between the predicted conversion labels and the ground truth data,using .By optimizing the conversion confusion loss, the gap between the transformed and untransformed,domains is minimized.

[0209] The auxiliary loss (conversion confusion loss) is the accuracy of the conversion prediction. For example, the predicted value is ["non-converted", "converted"], and the correct data label is ["non-converted", "converted"]. Therefore, the auxiliary loss = 1 (accuracy = 100%). However, we want to minimize this loss. In other words, we want the prediction to be wrong.

[0210] As shown in Figure 12, the target domain structural feature X T is input from the second feature extraction unit 112 to the auxiliary task solver 312. The converted source domain structural feature X ’T is input to the auxiliary task solver 312 from the second relighting unit 132.

[0211] The auxiliary task solver 312 calculates the target domain structural features X T Specifically, the auxiliary task solver 312 calculates the probability that each feature has been transformed by the second rigid body transformation unit 122 and the first relighting unit 132 and the probability that it has not been transformed, and uses the transformation label with the higher probability as the predicted value.

[0212] Auxiliary label data Y ’T is input to the auxiliary loss calculation unit 322. The auxiliary label data Y ’T is used to calculate the auxiliary loss. In this option, the auxiliary label data Y ’T For each feature, the correct transformed label data is either the "transformed domain" or the "untransformed domain." The auxiliary loss calculation unit 322 calculates the auxiliary label data Y ’TWe obtain the auxiliary loss, the conversion confusion loss (Loss_conversion_confusion_T), by calculating the correspondence between the predicted conversion labels and the conversion labels of the ground truth data using

[0213] In this option, the merge loss calculation unit 155 calculates the conversion loss as follows: In Equation 18, η″ and ξ′ are weighting coefficients.

[0214] Loss_merge = αLoss_classification_S + βLoss_classification_T + γLoss_grouping +δLoss_conversion + η''Loss_conversion_confusion_S + ξ'Loss_conversion_confusion_T (Equation 18)

[0215] Options 1 to 3 allow for higher quality feature extraction since not only the final classification goal but also the secondary goals are satisfied.

[0216] [Training equipment operation] Next, the operation of the training device 105 will be described with reference to the flowchart in Fig. 13. The processing of steps S100 to S154 and steps S156 to S158 is the same as the operation of the training device 104 of the third embodiment shown in Fig. 8. Note that the operation shown in Fig. 13 is mainly focused on the above-mentioned options 1 and 2.

[0217] In step S311, the auxiliary task solvers 311 and 312 generate auxiliary data. In option 1, the auxiliary data is a reconstructed image. In option 2, the auxiliary data is an angle prediction.

[0218] In step S312, the auxiliary loss calculation units 321 and 322 calculate the auxiliary losses. In option 1, the auxiliary losses are the source domain reconstruction loss and the target domain reconstruction loss. In option 2, the auxiliary losses are the source domain angle prediction loss and the target domain angle prediction loss.

[0219] In option 3, in step S311, the auxiliary task solvers 311 and 312 calculate the structural feature X S ,X T The auxiliary loss calculation units 321 and 322 calculate the conversion confusion loss.

[0220] In step S155C, the merge loss calculation unit 155 calculates the merge loss (Loss_merge) by referring to the source domain classification loss (Loss_classification_S), the target domain classification loss (Loss_classification_T), the grouping loss (Loss_grouping), the conversion loss (Loss_conversion), and the auxiliary loss. The merge loss calculation unit 155 calculates the merge loss using, for example, Equation 16, Equation 17, or Equation 18.

[0221] Embodiment 7. Fig. 14 is a block diagram showing an example of the configuration of a training device according to the seventh embodiment. The training device 104 shown in Fig. 10 has a configuration in which a first structural feature mask unit 411, a first conversion feature mask unit 421, a second structural feature mask unit 412, and a second conversion feature mask unit 422 are added to the training device 103 according to the third embodiment shown in Fig. 5.

[0222] This embodiment is based on the premise that structural features are represented by feature maps rather than by point coordinates. If structural features are represented as feature maps rather than by point coordinates, information about boundary areas will be lost after rigid body transformation. Therefore, to make it possible to compare the feature amount maps before and after transformation, masking is performed to remove information at the boundaries. This can improve classification performance.

[0223] Rigid transformation without masking results in missing information at the boundaries of the maps after transformation. Therefore, when directly comparing the untransformed feature map with the transformed feature map, these missing regions always result in large pixel-wise differences. By applying a mask to both the transformed and untransformed feature maps, we can eliminate the errors caused by the missing regions.

[0224] The first structural feature mask portion 411 is a source domain feature X S The second structure feature masking unit 412 masks the edge regions in the map of the target domain feature X. T The first transformed feature mask unit 421 masks edge regions in the map of transformed structural features from the first rigid transformation unit 121. The second transformed feature mask unit 422 masks edge regions in the map of transformed structural features from the second rigid transformation unit 122.

[0225] [Training equipment operation] Next, the operation of the training device 106 will be described with reference to the flowchart in Fig. 15. The processing of steps S100 to S111, the processing of steps S141 to S142, and the processing of steps S155 to S158 are the same as the operation of the training device 104 of the third embodiment shown in Fig. 8. In this embodiment, the processing of step S411 is executed before the processing of step S121, and the processing of step S421 is executed before the processing of step S131. The processing of step S412 is executed before the processing of step S122, and the processing of step S422 is executed before the processing of step S132.

[0226] In steps S411, S412, S421, and S422, the first structural feature mask unit 411, the second structural feature mask unit 412, the first converted feature mask unit 421, and the second converted feature mask unit 422 perform the above-mentioned mask processing on the edges of each map.

[0227] [Technical effect of this embodiment] In this embodiment, the error in the conversion loss calculated by the conversion loss calculation unit 154 can be reduced.

[0228] Each component in the above embodiment can be configured as a single piece of hardware, or as a single piece of software. Each component can also be configured as multiple pieces of hardware, or as multiple pieces of software. Furthermore, some of the components can be configured as hardware, and the other parts can be configured as software.

[0229] Each function (each process) in the above-described embodiments can be realized by a computer having a processor such as a CPU (Central Processing Unit), a memory, etc. For example, a program for implementing the method (process) in the above-described embodiments may be stored in a storage device (storage medium), and each function may be realized by executing the program stored in the storage device by a CPU.

[0230] FIG. 16 is a block diagram showing an example of a computer having a CPU. The computer is implemented in a training device and a classification device. The CPU 1000 executes processing in accordance with a program stored in a storage device 1001, thereby realizing the functions of the above-described embodiment. That is, the computer can implement the functions of the feature extraction unit 11, rigid body transformation unit 12, relighting unit 13, class prediction unit 14, and update unit 15 in the training device shown in FIG. 1 in accordance with the program.

[0231] By executing processing in accordance with the program, the computer can realize the functions of the first feature extraction unit 111, the second feature extraction unit 112, the first rigid body transformation unit 121, the second rigid body transformation unit 122, the first re-illumination unit 131, the second re-illumination unit 132, the first class prediction unit 141, the second class prediction unit 142, the update unit 150, the domain matching unit 211, the domain matching loss calculation unit 212, the auxiliary task solvers 311, 312, the auxiliary loss calculation units 321, 322, the first structural feature mask unit 411, the first transformation feature mask unit 421, the second structural feature mask unit 412, and the second transformation feature mask unit 422 in the training apparatus shown in Figures 5, 10, 12, and 14.

[0232] The computer can implement the functions of the feature extraction unit 61, rigid body transformation unit 62, relighting unit 63, and class prediction unit 64 in the classification device shown in FIGS. 2 and 9 by executing processes according to the program.

[0233] The storage device 1001 is, for example, a non-transitory computer readable medium. The non-transitory computer readable medium includes various types of tangible storage medium. Specific examples of non-transitory computer readable media include magnetic recording media (e.g., hard disks), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Compact Disc-Read Only Memory), CD-Rs (Compact Disc-Recordable), CD-R / Ws (Compact Disc-ReWritable), and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), and flash ROMs).

[0234] The program may also be stored in various types of transitory computer-readable media, to which the program is supplied, for example, via a wired or wireless communication path, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0235] The memory 1002 is realized by, for example, a random access memory (RAM), and is a storage means for temporarily storing data when the CPU 1000 executes processing. A configuration is also conceivable in which a program held in the storage device 1001 or a temporary computer-readable medium is transferred to the memory 1002, and the CPU 1000 executes processing based on the program in the memory 1002.

[0236] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0237] (Supplementary Note 1) One or more feature extraction means for extracting source domain structural features from input source domain image data and extracting target domain structural features from input target domain image data; a rigid body transformation means for performing rigid body transformation on the structural feature by referring to the transformation parameters to generate a transformed structural feature; one or more relighting means for generating features of a new view by referring to the transformed structural features and the transformation parameters so as to approximate structural features extracted from image data input at a view indicated by the transformation parameters; one or more class prediction means for performing source domain class prediction from the source domain structural features and source domain novel view features, and for performing target domain class prediction from the target domain structural features and target domain novel view features; and updating means for updating at least one of the one or more feature extraction means, the one or more relighting means, and the one or more class prediction means. Training equipment.

[0238] (Note 2) The update means executes the update process by referring to one or more of the following items: 10. The training device of claim 1. 1) A source domain classification loss calculated by referring to the predicted value of the source domain class calculated by the class prediction means and the correct class label of the source domain; 2) a target domain classification loss calculated by referring to the predicted value of the target domain class calculated by the class prediction means and the correct class label of the target domain; 3) A grouping loss calculated by referring to one or more features from the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features, and a class label corresponding to each feature; 4) A transformation loss calculated with reference to one or more features from the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features.

[0239] (Supplementary Note 3) A merge loss calculation means is provided to calculate a merge loss by referring to the source domain classification loss, the target domain classification loss, the grouping loss, and the conversion loss; The updating means updates at least one of the one or more feature extracting means, the one or more relighting means, and the one or more class predicting means if the merge loss has not converged. 10. The training device of claim 2.

[0240] (Supplementary Note 4) A classification loss calculation means is provided for calculating the source domain classification loss by referring to a source domain class prediction value, a source domain class prediction value of a feature of a new view of the source domain, and source domain class label data, and for calculating the target domain classification loss by referring to the target domain class prediction value, a target domain class prediction value of a structural feature after transformation, and target domain class label data. 10. The training device of claim 3.

[0241] (Supplementary Note 5) A grouping means for generating class groups including features that share the same class label from the source domain structural features, the source domain structural features after transformation, the target domain structural features, and the target domain structural features after transformation; a grouping loss calculation means for calculating the grouping loss by referring to the class groups generated by the grouping means; 10. The training device of claim 3 or 4.

[0242] (Supplementary Note 6) A conversion loss calculation means is provided for calculating the conversion loss by referring to one or more of the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features. 6. A training device according to any one of appendices 3 to 5.

[0243] (Appendix 7) A domain matching means for performing a domain matching process to match a target domain with a source domain; a domain matching loss calculation means for calculating a domain matching loss according to the distance between the source domain and the target domain; the merge loss calculation means calculates the merge loss by referring to the domain consistency loss; The updating means further updates the domain matching means. 7. A training device according to any one of claims 1 to 6.

[0244] (Appendix 8) Auxiliary task solvers to satisfy secondary goals in addition to the final classification goal, an auxiliary loss calculation means for calculating an auxiliary loss; the merge loss calculation means calculates the merge loss by referring to the auxiliary loss; The updating means further updates the auxiliary task solver. 7. A training device according to any one of claims 1 to 6.

[0245] (Supplementary Note 9) A structural feature masking means for masking edge regions in the map of source domain features; and a transformed feature masking means for masking edge regions in the transformed structural feature map. 7. A training device according to any one of claims 1 to 6.

[0246] (Appendix 10) A feature extraction means for extracting structural features from input image data; a class prediction means for predicting a class prediction value from the feature amount, At least one of the feature extraction means and the class prediction means is trained by referring to features of a new view obtained by transforming structural features. Classification device.

[0247] (Appendix 11) Using one or more feature extraction means, extract source domain structural features from the input source domain image data, and extract target domain structural features from the input target domain image data; generating transformed structural features by rigidly transforming the structural features with reference to the transformation parameters using one or more rigid transformation means; using one or more relighting means to generate features of a new view with reference to the transformed structural features and the transformation parameters so as to approximate structural features extracted from the input image data at a view indicated by the transformation parameters; using one or more class predictors to perform source domain class prediction from the source domain structural features and source domain novel view features, and to perform target domain class prediction from the target domain structural features and target domain novel view features; updating at least one of the one or more feature extractors, the one or more relighters, and the one or more class predictors; Training methods.

[0248] (Supplementary Note 12) When performing the update process, update at least one of the one or more feature extraction means, the one or more relighting means, and the one or more class prediction means with reference to one or more of the following: The training method described in Appendix 11. 1) A source domain classification loss calculated by referring to the predicted value of the source domain class calculated by the class prediction means and the correct class label of the source domain; 2) a target domain classification loss calculated by referring to the predicted value of the target domain class calculated by the class prediction means and the correct class label of the target domain; 3) A grouping loss calculated by referring to one or more features from the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features, and a class label corresponding to each feature; 4) A transformation loss calculated with reference to one or more features from the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features.

[0249] (Supplementary Note 13) Calculating a merge loss by referring to the source domain classification loss, the target domain classification loss, the grouping loss, and the transformation loss; If the merge loss has not converged, update at least one of the one or more feature extractors, the one or more relighters, and the one or more class predictors. The training method described in Appendix 12.

[0250] (Appendix 14) Extracting structural features from the input image data using feature extraction means; A class predictor is used to predict a class prediction value from the features; At least one of the feature extraction means and the class prediction means is trained by referring to features of a new view obtained by transforming structural features. Classification method.

[0251] (Appendix 15) A computer-readable recording medium storing a training program, The training program is configured to: extracting source domain structural features from the input source domain image data and target domain structural features from the input target domain image data using one or more feature extraction means; generating transformed structural features by rigidly transforming the structural features with reference to the transformation parameters using one or more rigid transformation means; using one or more relighting means to generate features of a new view with reference to the transformed structural features and the transformation parameters so as to approximate structural features extracted from the input image data at a view indicated by the transformation parameters; using one or more class prediction means to perform source domain class prediction from the source domain structural features and source domain novel view features, and to perform target domain class prediction from the target domain structural features and target domain novel view features; At least one of the one or more feature extractors, the one or more relighters, and the one or more class predictors is updated.

[0252] (Appendix 16) The training program is installed on a computer. When performing the update process, update at least one of the one or more feature extraction means, the one or more relighting means, and the one or more class prediction means with reference to one or more of the following: 16. The computer-readable storage medium of claim 15. 1) A source domain classification loss calculated by referring to the predicted value of the source domain class calculated by the class prediction means and the correct class label of the source domain; 2) a target domain classification loss calculated by referring to the predicted value of the target domain class calculated by the class prediction means and the correct class label of the target domain; 3) A grouping loss calculated by referring to the source domain structural features and their corresponding class labels, the transformed structural features of the source domain and their corresponding class labels, the target domain structural features and their corresponding class labels, and the transformed structural features of the target domain and their corresponding class labels; 4) A transformation loss calculated with reference to the source domain structural features, the transformed structural features of the source domain, the target domain structural features, and the transformed structural features of the target domain. 15. A recording medium as described in Appendix 14.

[0253] (Appendix 17) A computer-readable recording medium storing a classification program, The classification program is configured to extracting structural features from the input image data using a feature extraction means; A class prediction means is used to predict a class prediction value from the feature. At least one of the feature extraction means and the class prediction means is trained by referring to features of a new view obtained by transforming structural features.

[0254] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0255] 10, 103-106 Training equipment 11 Feature Extraction Unit 12 Rigid body transformation section 13 Re-lighting part 14 Class Prediction Unit 15 Update section 60,70 classification device 61 Feature Extraction Unit 62 Rigid body transformation part 63 Re-lighting section 64 Class Prediction Unit 111 First feature extraction unit 112 Second feature extraction unit 121 First rigid body transformation part 122 Second rigid body transformation part 131 1st re-illumination section 132 2nd re-illumination section 141 First Class Prediction Unit 142 Second Class Prediction Unit 150 Update section 151 Classification loss calculation section 152 Grouping Section 153 Grouping loss calculation unit 154 Conversion loss calculation section 155 Merge Loss Calculation Unit 156 Model Update Department 211 Domain Consistency 212 Domain Matching Loss Calculation Unit 311,312 Auxiliary Task Solver 321,322 Auxiliary loss calculation section 411 First structural feature mask part 412 Second structural feature mask part 421 First transformation feature mask part 422 Second transformation feature mask part

Claims

1. one or more feature extraction means for extracting source domain structural features from the input source domain image data and for extracting target domain structural features from the input target domain image data; a rigid body transformation means for performing rigid body transformation on the structural feature by referring to the transformation parameters to generate a transformed structural feature; one or more new view feature generating means for generating new view features by referring to the transformed structural features and the transformation parameters so as to approximate structural features extracted from image data input at a view indicated by the transformation parameters; one or more class predictors for predicting a source domain class prediction value from the source domain structural features and source domain novel view features, and predicting a target domain class prediction value from the target domain structural features and target domain novel view features; and updating means for updating at least one of the one or more feature extracting means, the one or more new view feature generating means, and the one or more class predicting means. Training equipment.

2. The update means performs the update process by referring to one or more of the following:

10. The training device of claim 1. 1) a source domain classification loss calculated by referring to the predicted value of the source domain class calculated by the class prediction means and the correct class label of the source domain; 2) a target domain classification loss calculated by referring to the predicted value of the target domain class calculated by the class prediction means and the correct class label of the target domain; 3) A grouping loss calculated with reference to one or more features from the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features, and a class label corresponding to each feature; 4) A transformation loss calculated with reference to one or more features from the source domain structural features, source domain novel view features, the target domain structural features, and target domain novel view features.

3. a merge loss calculation means for calculating a merge loss by referring to the source domain classification loss, the target domain classification loss, the grouping loss, and the conversion loss; The updating means updates at least one of the one or more feature extracting means, the one or more new view feature generating means, and the one or more class predicting means when the merge loss has not converged.

3. The training device of claim 2.

4. a classification loss calculation means for calculating the source domain classification loss by referring to source domain class prediction values ​​of source domain structural features, source domain class prediction values ​​of source domain novel view features, and source domain class label data, and for calculating the target domain classification loss by referring to target domain class prediction values ​​of target domain structural features, target domain class prediction values ​​of target domain novel view features, and target domain class label data; 4. The training device of claim 3.

5. a grouping means for generating class groups including feature quantities that share the same class label from the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features; a grouping loss calculation means for calculating the grouping loss by referring to the class groups generated by the grouping means; 5. The training device according to claim 3 or 4.

6. a conversion loss calculation means for calculating the conversion loss with reference to one or more of the source domain structural features, the source domain novel view features, the target domain structural features, and the target domain novel view features; A training device according to any one of claims 3 to 5.

7. domain matching means for performing a domain matching process to match a target domain with a source domain; a domain matching loss calculation means for calculating a domain matching loss according to the distance between the source domain and the target domain; the merge loss calculation means calculates the merge loss by referring to the domain consistency loss; The updating means further updates the domain matching means.

4. The training device of claim 3.

8. Auxiliary task solvers to satisfy secondary goals in addition to the final classification goal, an auxiliary loss calculation means for calculating an auxiliary loss; the merge loss calculation means calculates the merge loss by referring to the auxiliary loss; The updating means further updates the auxiliary task solver.

4. The training device of claim 3.

9. structural feature masking means for masking edge regions in the map of source domain features; and a transformation feature masking means for masking edge regions in the map of features of the new view. A training device according to any one of claims 1 to 6.

Images