Information processing device, information processing method, and program

Abstract

In order to solve the problem of generating an attention map independent on a domain of data being processed, this information processing device comprises: an acquisition means (21) which acquires data belonging to a target domain; a first feature value extraction means (22) which extracts a first feature value from the data; an attention map generation means (23) which generates an attention map from the first feature value and which is configured so as to attenuate domain-dependency of the attention map generated; and an identification means (24) which carries out data identification processing on the basis of the first feature value and the attention map.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] The technology of classifying target data using models obtained by machine learning is widely used. For example, there is a demand for a system that classifies the models of cars captured in images taken by surveillance cameras installed on roads.

[0003] When there are many types of target data and only a small amount of visual difference, it is preferable to use attention maps to perform classification processing in order to improve classification accuracy. For example, it is possible to use a model obtained by machine learning to predict attention maps in images captured by a surveillance camera.

[0004] To predict the attention map, a large number of pre-labeled images must be prepared and used to train the model, but it is difficult to obtain a large number of real-life images.

[0005] In such cases, for example, it is possible to use a CG (computer graphics) image obtained by rendering a CG model as a labeled image with an attention map attached, and to use domain adaptation technology in which the CG image is the source domain and the live-action image is the target domain.

[0006] Patent Document 1 discloses a technique for creating training data for a neural network using CG images, etc. Non-Patent Document 2 discloses an adversarial domain adaptation technique. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2021-18477 [Non-patent literature]

[0008] [Non-Patent Document 1] Y Ganin and V Lempitsky, Unsupervised Domain Adaptation by Backpropagation, arXiv:1409.7495 [Non-patent document 2] Saeid Motiian, Marco Piccirilli, Donald A. Adjeroh, Gianfranco Doretto, Unified Deep Supervised Domain Adaptation and Generalization, https: / / arxiv.org / pdf / 1709.10190v1.pdf Summary of the Invention [Problem to be solved by the invention]

[0009] The inventors have found that even if classification processing using attention maps is combined with conventional domain adaptation technology, domain dependency is likely to remain in the attention map, which poses problems in terms of classification accuracy in the target domain.

[0010] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its purpose is to provide a technology that can perform classification processing for a target domain in an appropriate manner while using an attention map. [Means for solving the problem]

[0011] An information processing device according to one aspect of the present invention comprises an acquisition means for acquiring data belonging to a target domain, a first feature extraction means for extracting a first feature from the data, an attention map generation means for generating an attention map from the first feature, the attention map generation means being configured to reduce the domain dependency of the generated attention map, and an identification means for performing an identification process on the data based on the first feature and the attention map.

[0012] An information processing device according to one aspect of the present invention includes acquiring data belonging to a target domain, extracting a first feature from the data, generating an attention map from the first feature such that the domain dependency of the generated attention map is reduced, and performing an identification process on the data based on the first feature and the attention map.

[0013] A program according to one aspect of the present invention causes a computer to function as an information processing device comprising: an acquisition means for acquiring data belonging to a target domain; a first feature extraction means for extracting a first feature from the data; an attention map generation means for generating an attention map from the first feature, the attention map generation means being configured to reduce the domain dependency of the generated attention map; and an identification means for performing an identification process on the data based on the first feature and the attention map.

[0014] An information processing device according to another aspect of the present invention comprises an acquisition means for acquiring data belonging to a source domain and label data of an attention map associated with the data, a feature extraction means for extracting a first feature from the data, an attention map generation means for generating an attention map from the feature of the data and the first feature, and a learning means for training the attention map generation means using the label data, the attention map generation means being configured to reduce the domain dependency of the attention map to be generated.

[0015] An information processing method according to another aspect of the present invention includes acquiring data belonging to a source domain and label data of an attention map associated with the data, extracting a first feature from the data, and training an attention map generation means that generates an attention map from the feature of the data and the first feature, the attention map generation means being configured to reduce the domain dependency of the attention map to be generated, using the label data.

[0016] A program relating to another aspect of the present invention causes a computer to function as an information processing device having an acquisition means for acquiring data belonging to a source domain and label data of an attention map associated with the data, a feature extraction means for extracting a first feature from the data, and an attention map generation means for generating an attention map from the feature of the data and the first feature, wherein the attention map generation means is configured to reduce the domain dependency of the generated attention map, and a learning means for training the attention map generation means using the label data. [Effects of the Invention]

[0017] According to one aspect of the present invention, an attention map can be generated that is independent of the domain of the target data. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 2] 1 is a flowchart showing the flow of an information processing method according to the first exemplary embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of another information processing device according to the first exemplary embodiment of the present invention. [Figure 4] 1 is a flowchart showing the flow of a learning process according to the first exemplary embodiment of the present invention. [Figure 5]FIG. 10 is a diagram illustrating a prediction method used by an information processing device according to a second exemplary embodiment of the present invention. [Figure 6] FIG. 10 is a block diagram showing an example of the configuration of an information processing device according to a second exemplary embodiment of the present invention. [Figure 7] FIG. 10 is a diagram illustrating a specific configuration example of an attention map generating unit in exemplary embodiment 2, together with the peripheral configuration. [Figure 8] FIG. 8 is a diagram illustrating a detailed configuration example of a domain-independent channel attention extractor in FIG. 7. [Figure 9] 9 is a diagram illustrating a detailed configuration example of a domain classification unit in FIG. 8. FIG. [Figure 10] 10 is a flowchart illustrating the flow of an identification process. [Figure 11] 10 is a flowchart illustrating the flow of an attention map generation process. [Figure 12] 10 is a flowchart illustrating the flow of a channel attention generation process. [Figure 13] FIG. 10 is a block diagram showing an example of the configuration of an information processing device according to a third exemplary embodiment of the present invention. [Figure 14] 10A and 10B are diagrams illustrating generation of a CG image with an attention map from a CG model. [Figure 15] 14 is a block diagram showing a detailed configuration example of a label data generating unit in FIG. 13. FIG. [Figure 16] 16 is a block diagram showing a detailed configuration example of an attention-attached rendering processing unit in FIG. 15. FIG. [Figure 17] FIG. 16 is a block diagram showing another example of the detailed configuration of the attention-attached rendering processing unit of FIG. 15. [Figure 18] 10 is a flowchart illustrating the flow of an attention map-attached image generation process. [Figure 19] 10 is a flowchart illustrating the flow of a rendering process A. [Figure 20] 10 is a flowchart illustrating the flow of a rendering process B. [Figure 21]FIG. 1 is a diagram illustrating an example of the configuration of a computer that executes instructions of a program, which is software that realizes each function. DETAILED DESCRIPTION OF THE INVENTION

[0019] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0020] <Overview of information processing device 20> The information processing device 20 according to this exemplary embodiment is, in brief, a device that uses an attention map to identify target data, and performs domain adaptation using a source domain and a target domain. As an example, the information processing device 20 is a device that performs detailed classification of target data types with little difference. The information processing device 20 classifies target data belonging to the target domain using a model obtained by machine learning. Here, the classification process using the model includes processing that references the attention map. In addition, the above machine learning includes, as an example, A supervised classification learning process using a set of source domain data, which is data belonging to a source domain, and class label data associated with the source domain data; and Domain adaptation processing using a pair of target domain data, which is data belonging to a target domain, and domain label data associated with the target domain data, and a pair of source domain data and domain label data associated with the source domain data may be included.

[0021] Here, the class label data is label data indicating which classification destination (class) the data associated with the class label data (source domain data in the above example) should be classified into. Also, the domain label data is label data indicating which domain the data associated with the domain label data (source domain data or target domain data in the above example) belongs to.

[0022] In a classification learning process using class label data, for example, the learning process is performed so that the difference between the class output by the model and the class indicated by the label data becomes small.

[0023] In addition, in the domain adaptation process using domain label data, as an example, the domain adaptation process is performed so that the difference between the classification process for the source domain data and the classification process for the target domain becomes small.

[0024] In other words, in a domain adaptation process using domain label data, the domain adaptation process is performed in such a way that it becomes difficult for the model to distinguish whether it is performing a classification process on source domain data or a classification process on a target domain, for example.

[0025] Here, the source domain data is data that, for example, facilitates training of the attention map generator that generates the attention map compared to the target domain data. Furthermore, the source domain has a wealth of data compared to the target domain. Therefore, by using a model that has undergone the above-described classification learning process and domain adaptation process, it is expected that the benefits of learning from the data-rich source domain can be enjoyed while the classification process for the target domain can be suitably performed.

[0026] Regarding the above point, the inventor has found that even if conventionally known processes such as the class classification learning process and domain adaptation process described above are performed, domain dependency is likely to remain in the attention map, and therefore there are areas that need to be improved in terms of classification accuracy in the target domain.

[0027] The information processing device 20 according to this exemplary embodiment has been developed in consideration of the above-mentioned problems. As an example, an acquisition means for acquiring data belonging to a target domain; a first feature extraction means for extracting a first feature from the data; an attention map generating means for generating an attention map from the first feature amount, the attention map generating means being configured to reduce domain dependency of the generated attention map; The apparatus further comprises a classification means for performing a classification process on the data based on the first feature amount and the attention map.

[0028] According to the above configuration, by using attention map generating means configured to reduce the domain dependency of the attention map, it is possible to suitably improve classification accuracy in the target domain.

[0029] <Configuration of information processing device 20> The configuration of the information processing device 20 according to this exemplary embodiment will be described in more detail below with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the information processing device 20.

[0030] 1, the information processing device 20 includes an acquisition unit 21, a feature extraction unit 22, an attention map generation unit 23, and a classification unit 24. The acquisition unit 21 is configured to realize an acquisition means in this exemplary embodiment. The feature extraction unit 22 is configured to realize a first feature extraction means in this exemplary embodiment. The attention map generation unit 23 and the classification unit 24 are configured to realize an attention map generation means and a classification means, respectively, in this exemplary embodiment.

[0031] The acquisition unit 21 acquires input data belonging to the target domain. Hereinafter, unless there is any particular confusion, the input data may be simply referred to as data. In this exemplary embodiment, when simply referred to as data or input data, this may include data belonging to the target domain and data belonging to the source domain. The data acquired by the acquisition unit 21 is, for example, image data, and the data belonging to the target domain may be, for example, live-action image data. Also, the data belonging to the source domain may be, for example, CG image data, but is not limited to these. Furthermore, the data acquired by the acquisition unit is not limited to image data.

[0032] In this exemplary embodiment, data belonging to the target domain is also referred to as target domain data, and data belonging to the source domain is also referred to as source domain data.

[0033] The feature extraction unit 22 extracts a first feature from the target data. As an example, the feature is a feature that is referenced to generate an attention map for the data acquired by the acquisition unit 21, and is also a feature that is referenced to classify subjects included in an image indicated by the data acquired by the acquisition unit 21. When the data acquired by the acquisition unit 21 is an image, the feature extracted by the feature extraction unit 22 can also be expressed as one or more two-dimensional maps. For this reason, in this exemplary embodiment, the feature is sometimes referred to as a feature map.

[0034] The feature extraction unit 22 and the identification unit 24 described later can use data that has been pre-trained (class classification learning and domain adaptation) using training data that includes multiple pairs of data acquired by the acquisition unit 21 and label data (class label data and domain label data).

[0035] The attention map generation unit 23 generates an attention map from the first feature extracted by the feature extraction unit 22. The attention map is a map that indicates points that should be paid attention to in the feature extracted from the data (feature map) when the data is classified by the classification unit 24.

[0036] For example, consider a case where the data acquired by the acquisition unit 21 is an image of an automobile, particularly a passenger car. There are an extremely large number of vehicle models, and some models are difficult to distinguish at a glance. In such cases, it may be impossible to identify the model without comparing, for example, the shape of the front grille or the shape of the fenders. When identifying the data acquired by the acquisition unit 21, the accuracy and efficiency of the identification can be improved by, for example, referring to an attention map that includes parts such as the front grille and the fenders as attention points.

[0037] The attention map generating unit 23 generates an attention map. Here, the attention map generating unit 23 is configured to reduce domain dependency.

[0038] As an example, the processing by the attention map generating unit 23 includes the following steps: A classification process for classifying domains based on the first feature amount; and A process of converting the classification result obtained by the classification process and multiplying it by the weighting coefficient used in the classification process. By having the above configuration, the attention map generating unit 23 can reduce the domain dependency of the attention map.

[0039] More specifically, as an example, the processing by the attention map generating unit 23 includes the following steps: A process of extracting a second feature amount from the first feature amount or a feature amount obtained from the first feature amount. A classification process for performing domain classification based on the second feature amount. A process of calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classification means and the classification result by the classification means; and A process of calculating the attention map using the first feature amount or a feature amount obtained from the first feature amount and the channel attention. may be included.

[0040] The process of calculating the channel attention includes: -Conversion process to convert the classification results, and A process of calculating the channel attention using the weighting coefficient and the classification result after the conversion process. may be included.

[0041] In this way, the attention map generation unit 23 can, for example, reduce the domain dependency of the attention map by calculating the channel attention using the weighting coefficient and the classification result after the conversion process.

[0042] The conversion process includes the following steps: A process of exchanging the confidence levels of each of the multiple domains in the classification result; or A process for reducing the difference in the confidence levels of each of the multiple domains in the classification results. It is preferred that it contains: By having the attention map generating unit 23 have the above configuration, it is possible to more suitably reduce the domain dependency of the attention map.

[0043] If the domain dependency is weakened, it is expected that the classification accuracy will be improved by applying the generated attention map regardless of the domain to which the data acquired by the acquisition unit 21 belongs. As an example, even if an attention map generated based on features extracted from a CG image is applied to a real-life image to classify the data acquired by the acquisition unit 21, it will be possible to achieve favorable classification accuracy.

[0044] The attention map generating unit 23 is trained in advance using training data including a plurality of pairs of input data or feature maps extracted from the input data and label data (attention map labels) of the attention map corresponding to the input data. As an example, the attention map generating unit 23 is trained in advance so that the difference between the attention map generated by the attention map generating unit 23 and the label data of the attention map is small.

[0045] Furthermore, the attention map generating unit 23 may be domain-adapted in advance using training data including a plurality of pairs of input data and domain label data associated with the input data.

[0046] The identification unit 24 refers to the attention map generated by the attention map generation unit 23 and performs an identification process on the data acquired by the acquisition unit 21 based on the first feature extracted by the feature extraction unit 22. The identification unit 24 identifies, for example, the type of car shown in the image of the data acquired by the acquisition unit 21.

[0047] <Flow of information processing method by information processing device 20> The flow of the information processing method executed by the information processing device 20 configured as above will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the flow of the information processing method. As shown in the figure, the information processing includes steps S11 to S14.

[0048] In step S11, the acquisition unit 21 acquires data belonging to the target domain.

[0049] In step S12, the feature extraction unit 22 extracts a first feature from the data acquired in step S11.

[0050] In step S13, the attention map generation unit 23 generates an attention map from the first feature amount extracted in the process of step S12. At this time, the attention map generation unit 23 generates the attention map so that the domain dependency of the generated attention map is weakened.

[0051] As an example, the process of step S13 includes the following steps: A classification process for classifying domains based on the first feature amount; and A process of converting the classification result obtained by the classification process and multiplying it by the weighting coefficient used in the classification process. The above process can reduce the domain dependency of the attention map.

[0052] More specifically, as an example, the process of step S13 includes the following steps: A process of extracting a second feature amount from the first feature amount or a feature amount obtained from the first feature amount. A classification process for performing domain classification based on the second feature amount. A process of calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classifying means and the classification result by the classifying means; and A process of calculating the attention map using the first feature amount or a feature amount obtained from the first feature amount and the channel attention. may be included.

[0053] The process of calculating the channel attention includes: -Conversion process to convert the classification results, and A process of calculating the channel attention using the weighting coefficient and the classification result after the conversion process. may be included.

[0054] In this way, the attention map generation unit 23 can, for example, reduce the domain dependency of the attention map by calculating the channel attention using the weighting coefficient and the classification result after the conversion process.

[0055] The conversion process includes the following steps: A process of exchanging the confidence levels of each of the multiple domains in the classification result; or A process for reducing the difference in the confidence levels of each of the multiple domains in the classification results. It is preferred that it contains:

[0056] In step S14, the classification unit 24 performs a classification process on the target data based on the first feature amount extracted in step S12 and the attention map generated in step S13.

[0057] In this way, information processing is performed to identify the data, for example, to identify the model of the passenger car that is the subject of the image indicated by the data.

[0058] <Effects of the information processing device 20 and the information processing method> According to the information processing device 20 of this exemplary embodiment, data belonging to a target domain is acquired, features are extracted from the data, an attention map is generated from the features, and data classification processing is performed based on the features and the attention map. Since the attention map is generated to reduce domain dependency, improved classification accuracy can be expected by applying the generated attention map regardless of the domain to which the data belongs. As an example, even if a model obtained by supervised learning using mainly source domain data is used to classify target domain data, favorable classification accuracy can be achieved. Therefore, classification processing for the target domain can be performed favorably while using the attention map.

[0059] <Overview of information processing device 40> The information processing device 40 according to this exemplary embodiment is, generally speaking, a device that performs machine learning of a model used by the information processing device 20. This machine learning is mainly supervised learning using source domain data.

[0060] More specifically, the information processing device 40, for example, an acquisition means for acquiring data belonging to a source domain and label data of an attention map associated with the data; feature extraction means for extracting a first feature from the data; The attention map generation means generates an attention map from the features of the data and the first features, and is configured to reduce the domain dependency of the attention map to be generated, and the attention map generation means is equipped with a learning means that uses label data to train the attention map generation means.

[0061] <Configuration of information processing device 40> The configuration of the information processing device 40 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the information processing device 40.

[0062] 3, the information processing device 40 includes an acquisition unit 41, a feature extraction unit 42, an attention map generation unit 43, and a learning unit 44. The acquisition unit 41 is configured to realize acquisition means in this exemplary embodiment. The feature extraction unit 42 is configured to realize feature extraction means in this exemplary embodiment. The attention map generation unit 43 and the learning unit 44 are configured to realize attention map generation means and learning means, respectively, in this exemplary embodiment.

[0063] The acquisition unit 41 acquires data belonging to a source domain and label data of an attention map associated with the data. For example, the data is image data, and the data belonging to the source domain is CG image data. For example, the data belonging to the target domain is live-action image data.

[0064] The attention map label data serves as training data for machine learning. For example, the attention map label data is a CG image obtained by rendering a CG model with designated attention areas from multiple camera angles.

[0065] The feature extraction unit 42 extracts features from the data and is a functional block similar to the feature extraction unit 22 included in the information processing device 20 described above with reference to FIG.

[0066] The attention map generating unit 43 generates an attention map from the feature amount. The attention map generating unit 43 is a functional block similar to the attention map generating unit 23 included in the information processing device 20 described above with reference to FIG.

[0067] The learning unit 44 uses the label data to learn model parameters used by the feature extraction unit 42 and the attention map generation unit 43. The learning unit 44 uses the label data to learn so as to reduce the domain dependency of the attention map generated by the attention map generation unit 43. That is, the learning unit 44 uses the label data to train the attention map generation unit 43, which generates an attention map from the features of the data and the first feature, and is configured so as to reduce the domain dependency of the attention map to be generated.

[0068] The attention map generating unit 43 a second feature amount extraction unit that extracts a second feature amount from the first feature amount; a classification unit that performs domain classification based on the second feature amount; a channel attention calculation unit that calculates a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classification means and a classification result by the classification means; The learning unit 44 performs the following in the learning process of the attention map generation unit: without updating the weighting coefficient of the second feature amount extraction means, updating the weighting coefficients of the classifier; The following configuration may be adopted.

[0069] <Flow of information processing method by information processing device 40> In step S31, the acquisition unit 41 acquires data belonging to the source domain and label data of the attention map associated with the data.

[0070] In step S32, the feature extraction unit 42 extracts a first feature from the data.

[0071] In step S33, the learning unit 44 uses the label data to train the attention map generation unit 43, which is configured to reduce the domain dependency of the attention map generated from the data features and the first features.

[0072] <Effects of the information processing device 40 and the information processing method> According to the information processing device 40 of this exemplary embodiment, data belonging to a source domain and label data of an attention map associated with the data are acquired, a first feature is extracted from the data, and learning is performed to generate an attention map from the feature of the data and the first feature. During this process, learning is performed using the label data so as to reduce the domain dependency of the generated attention map. Because learning is performed to reduce domain dependency, it is possible to achieve favorable classification accuracy even when a model obtained by supervised learning using mainly source domain data is used to identify target domain data. Therefore, classification processing for the target domain can be performed favorably while using the attention map.

[0073] Exemplary Embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Fig. 5 is a diagram illustrating a prediction method by an information processing device in this exemplary embodiment.

[0074] In Fig. 5, the upper part shows a classification result 56 for a real-life image 51, and the lower part shows a classification result 76 for a CG image 71. The upper part of Fig. 5 is the domain of the real-life image, which corresponds to the target domain. The lower part of Fig. 5 is the domain of the CG image, which corresponds to the source domain.

[0075] In this exemplary embodiment, it is assumed that CG images 71 are more abundant and easier to obtain than real-life images 51. Therefore, in this exemplary embodiment, real-life images 51 can be identified by using a prediction model that has undergone classification learning processing and domain adaptation processing using CG images 71.

[0076] 5, the feature extractor 52 and the feature extractor 72 correspond to the feature extraction unit 22 in the exemplary embodiment 1. Furthermore, the attention extractor 54 and the attention extractor 74 correspond to the attention map generation unit 23 in the exemplary embodiment 1. Furthermore, the classifier 55 and the classifier 75 correspond to the identification unit 24 in the exemplary embodiment 1. Although not shown in the figure, the acquirers that acquire the CG image 71 and the image 51 and supply them to the feature extractor 52 and the feature extractor 72, respectively, correspond to the acquisition unit 21 in the exemplary embodiment 1.

[0077] 5, a CG image 71 is input to a feature extractor 72, which outputs a feature map 73. The feature map 73 is supplied to an attention extractor 74 and a classifier 75.

[0078] The attention extractor 74 extracts an attention map from the feature map 73. The attention map is a map that indicates areas in the feature map that should be paid attention to when the feature map 73 is classified by the classifier 75. The attention map extracted by the attention extractor 74 is supplied to the classifier 75.

[0079] The classifier 75 refers to the attention map and classifies the feature map 73. That is, the classifier 75 classifies the CG image 71 corresponding to the feature map 73 into one of the classes learned by the classification learning process. As a result, a classification result 76 for the CG image 71 is output.

[0080] To enable such processing, the feature extractor 72, attention extractor 74, and classifier 75 are trained. As an example, the feature extractor 72, attention extractor 74, and classifier 75 are each configured as a prediction model having a CNN (Convolutional Neural Network) structure.

[0081] That is, in the class classification learning process using the CG image 71 and class label data, the learning process is executed so that the difference between the class output by the model and the class indicated by the label data becomes small, for example.

[0082] Furthermore, to enable the above-described processing, a domain adaptation process is performed using a pair of a real-life image and domain label data associated with the real-life image, and a pair of a CG image and domain label data associated with the CG image. In the domain adaptation process using the domain label data, the domain adaptation process is performed so that the difference between the classification process for the CG image and the classification process for the real-life image is reduced.

[0083] In other words, the domain adaptation process is performed so that it becomes difficult for the predictive model to distinguish whether it is performing classification processing on a CG image or a real-life image.

[0084] Even if conventional classification learning and domain adaptation processes are performed, domain dependency is likely to remain in the attention map. Therefore, in order to improve the accuracy of the classification results for the target domain, i.e., the real-life image 51, it is necessary to reduce the domain dependency of the attention map extracted by the attention extractor 74. The method for reducing the domain dependency of the attention map will be described in detail later.

[0085] By using a prediction model that has undergone such classification and domain adaptation processing, it becomes possible to classify real-life images in the same way as CG images, which are the source domain.

[0086] 5 is input to a feature extractor 52, which outputs a feature map 53. The feature map 53 is supplied to an attention extractor 54 and a classifier 55.

[0087] The attention extractor 54 extracts an attention map from the feature map 53. The attention map extracted by the attention extractor 54 is provided to the classifier 55.

[0088] The classifier 55 refers to the attention map and classifies the feature map 53. That is, the classifier 55 classifies the real-life image 51 corresponding to the feature map 53 into one of the classes learned by the classification learning process. As a result, a classification result 56 for the real-life image 51 is output.

[0089] For ease of explanation, functional blocks related to the target domain are shown in the upper row in FIG. 5, and functional blocks related to the source domain are shown in the lower row. However, corresponding processes in both domains may be performed by a single functional block. Specifically, feature extractor 52 and feature extractor 72 may be configured as a single functional block. Similarly, attention extractor 54 and attention extractor 74, and classifier 55 and classifier 75 may each be configured as a single functional block.

[0090] <Configuration of information processing device 100> 6 is a block diagram showing an example of the configuration of an information processing device 100 according to this exemplary embodiment. As shown in FIG. 6, the information processing device 100 includes a control unit 101, an input unit 161, a communication unit 162, an output unit 163, and a storage unit.

[0091] The input unit 161 accepts various inputs to the information processing device 100. The specific configuration of the input unit 161 does not limit the present exemplary embodiment, but as an example, the input unit 161 may be configured to include input devices such as a keyboard and a touchpad. The input unit 161 may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays or radio waves, and a sensor that senses the environmental state.

[0092] The communication unit 162 is an interface for connecting the information processing device 100 to a network. The specific configuration of the network does not limit the present exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0093] The output unit 163 is a functional block that outputs processing results by the information processing device 100. The specific configuration of the output unit 163 does not limit this exemplary embodiment, but as an example, the output unit 163 is configured by a display, a speaker, a printer, etc., and displays various processing results by the information processing device 100 on a screen or outputs them as sounds or figures.

[0094] The storage unit stores, as an example, target domain data 111, source domain data 112, model parameters 113, and attention map labels 114 shown in FIG.

[0095] (Control unit 101) The control unit 101 will be described in detail below. As shown in Fig. 6, the control unit 101 includes components indicated by the reference numerals 21 to 24 and 44. In Fig. 6, the components indicated by the dotted line frames 21 to 24 correspond to the acquisition unit 21 to the identification unit 24 of the information processing device 20 shown in Fig. 1, respectively.

[0096] The information processing device in Fig. 6 is configured to be able to identify target domain data by using a prediction model that has undergone classification learning processing and domain adaptation processing using source domain data. As an example, the source domain data is image data of a CG image, and the target domain data is image data of a real-life image.

[0097] In the following, a case will be described as an example in which the information processing device 100 identifies the vehicle type shown in a CG image and a real image.

[0098] 6, the target domain data 111 and the source domain data 112 are data sets each storing a plurality of data. The target domain data 111 stores a plurality of image data of real-life images of automobiles, and the source domain data 112 stores a plurality of image data of CG images of automobiles.

[0099] The data acquisition unit 121A acquires image data from the target domain data 111. The data acquisition unit 121B acquires image data from the source domain data 112. The data acquisition unit 121A and the data acquisition unit 121B together correspond to the acquisition unit 21 and the acquisition unit 41 described in the first exemplary embodiment.

[0100] The feature extraction unit 122A (an example of a first feature extraction means) extracts a first feature from a real-life image, which is image data acquired by the data acquisition unit 121A. The feature extraction unit 122B (an example of a first feature extraction means) extracts a first feature from a CG image, which is image data acquired by the data acquisition unit 121B. The feature extraction units 122A and 122B together correspond to the feature extraction units 22 and 42 described in the first exemplary embodiment.

[0101] The attention feature pre-processing unit 131A executes pre-processing for generating an attention map from a first feature amount related to the real-life image. Specifically, the attention feature pre-processing unit 131A extracts further feature amounts related to the real-life image from the first feature amount related to the real-life image.

[0102] The attention feature pre-processing unit 131B executes pre-processing for generating an attention map from a first feature amount related to the CG image. Specifically, the attention feature pre-processing unit 131B extracts a further feature amount related to the CG image from the first feature amount related to the CG image.

[0103] The attention map feature extraction unit 133A generates a preliminary attention map for the real image from the additional feature amount for the real image generated by the attention feature pre-processing unit 131A.

[0104] The attention map feature extraction unit 133B generates a preliminary attention map for the CG image from the above-mentioned additional feature amount for the CG image generated by the attention feature pre-processing unit 131B.

[0105] The domain-independent channel attention calculation unit 132A calculates domain-independent channel attention for the real-life image from the additional feature amount for the real-life image generated by the attention feature pre-processing unit 131 A. A specific configuration of the domain-independent channel attention calculation unit 132A will be described later.

[0106] The domain-independent channel attention calculation unit 132B calculates domain-independent channel attention for the CG image from the additional feature amount for the CG image generated by the attention feature pre-processing unit 131B. A specific configuration of the domain-independent channel attention calculation unit 132B will be described later.

[0107] The domain-independent attention map generating unit 134A generates an attention map for the real-life image from the domain-independent channel attention for the real-life image calculated by the domain-independent channel attention calculating unit 132A and the preliminary attention map for the real-life image generated by the attention map feature extracting unit 133A. The generated attention map for the real-life image is supplied to the previous-stage feature identifying unit 141A together with the first feature amount for the real-life image.

[0108] The domain-independent attention map generation unit 134B generates an attention map for the CG image from the domain-independent channel attention for the CG image calculated by the domain-independent channel attention calculation unit 132B and the preliminary attention map for the CG image generated by the attention map feature extraction unit 133B. The generated attention map for the CG image is supplied to the previous-stage feature identification unit 141B together with the first feature amount for the CG image.

[0109] The attention feature front-end processing units 131A, 131B, domain-independent channel attention calculation units 132A, 132B, attention map feature extraction units 133A, 133B, and domain-independent attention map generation units 134A, 134B collectively correspond to the attention map generation unit 23 and the attention map generation unit 43 described in exemplary embodiment 1.

[0110] The former feature identification unit 141A generates a feature amount related to the real image from a first feature amount related to the real image and an attention map related to the real image, and supplies the generated feature amount to the latter feature identification unit 142.

[0111] The former feature identification unit 141B generates a feature related to the CG image from a first feature related to the CG image and an attention map related to the CG image, and supplies the generated feature to the latter feature identification unit 142.

[0112] The latter-stage feature classification unit 142 performs classification processing (classification processing) on ​​the CG image and the real image based on the feature amounts supplied from the former-stage feature classification unit 141A and the former-stage feature classification unit 141B, and outputs the classification results. As an example, the latter-stage feature classification unit 142 outputs the classification results of the car models shown in the CG image and the real image.

[0113] The front-stage feature identifying section 141A, the front-stage feature identifying section 141B, and the back-stage feature identifying section 142 correspond to the identifying section 24 described in the first exemplary embodiment.

[0114] The feature identification optimization unit 151 trains the preceding feature identification unit 141A, the preceding feature identification unit 141B, and the subsequent feature identification unit 142. The feature identification optimization unit 151 may also be configured to further train at least one of the series of components from the feature extraction unit 122A to the domain-independent attention map generation unit 134A. The feature identification optimization unit 151 may also be configured to further train at least one of the series of components from the feature extraction unit 122B to the domain-independent attention map generation unit 134B.

[0115] The feature identification optimization unit 151 trains the above-mentioned components so as to minimize the discrepancy between the classification result by the subsequent feature identification unit 142 for an image acquired by the data acquisition unit 121A or the data acquisition unit 121B and the class label assigned to the image. As an example, a class classification learning process is executed using a CG image obtained by rendering a CG model of an automobile and class label data indicating the type of automobile associated with the CG image. For example, the learning process by the feature identification optimization unit 151 can use backpropagation technology, but this is not intended to limit the present exemplary embodiment.

[0116] The domain adaptation unit 152 performs domain adaptation processing on the preceding feature identification unit 141A and the preceding feature identification unit 141B. The domain adaptation unit 152 may further perform domain adaptation processing on at least one of the components from the feature extraction unit 122A to the domain-independent attention map generation unit 134A and at least one of the components from the feature extraction unit 122B to the domain-independent attention map generation unit 134B.

[0117] The domain adaptation process by the domain adaptation unit 152 is performed, for example, by referring to a domain label associated with the image acquired by the data acquisition unit 121A or the data acquisition unit 121B.

[0118] As an example, the domain adaptation unit 152 performs domain adaptation processing on the above-mentioned units so as to reduce the difference between the classification processing on real images by the front-stage feature identification unit 141A and the classification processing on CG images by the front-stage feature identification unit 141B.

[0119] The specific domain adaptation algorithm executed by the domain adaptation unit 152 does not limit this exemplary embodiment, but as an example, the techniques described in Unsupervised Domain Adaptation by Backpropagation (Non-Patent Document 1) and Unified Deep Supervised Domain Adaptation and Generalization (Non-Patent Document 2) can be used.

[0120] The attention map optimization unit 153 trains at least one of the components from the attention feature preprocessing unit 131B to the domain-independent attention map generation unit 134B. The attention map optimization unit 153 performs a training process on each of the components described above to minimize the discrepancy between the attention map generated by the domain-independent attention map generation unit 134B for the CG image acquired by the data acquisition unit 121B and the label data (attention map label) of the attention map associated with the CG image. As an example, the attention map optimization unit 153 trains the domain-independent attention map generation unit 134B by referring to the label data (attention map label).

[0121] Note that the attention map optimization unit 153 may be configured to update the weighting coefficients of the domain classification unit 192 (classification means) described later, without updating the weighting coefficients of the CNN 191 (second feature extraction means) described later, in the learning process of the domain-independent attention map generation unit 134B. This configuration allows the domain-independent attention map generation unit 134B to learn more efficiently.

[0122] The attention map label data is, for example, an attention map extracted in advance from a CG image of the source domain data 112. A plurality of pieces of attention map label data are generated corresponding to a plurality of CG images of the source domain data 112, and are stored in the attention map label 114.

[0123] 6, each unit indicated by a dotted line frame 21 to 24 is configured as, for example, a prediction model having a CNN structure. These prediction models use parameters included in the model parameters 113 to execute each process.

[0124] (Attention map generation part) 7 is a diagram illustrating a specific configuration example of the attention map generation unit 23 in this exemplary embodiment, together with the peripheral configuration. As shown in Fig. 7, the attention map generation unit 23 includes a convolutional layer 211, a convolutional layer 212, a domain-independent channel attention extractor 190, and an integration unit 202. Furthermore, as shown in Fig. 7, the attention map generation unit 23 may also include other convolutional layers and a GAP (Global Average Pooling) layer.

[0125] The convolutional layer 211 corresponds to the attention feature preprocessing units 131A and 131B shown in Fig. 6. The convolutional layer 212 corresponds to the attention map feature extraction units 133A and 133B shown in Fig. 6. The integration unit 202 corresponds to the domain-independent attention map generation units 134A and 134B shown in Fig. 6.

[0126] In FIG. 7, the processing result from the convolutional layer 211 is output to the convolutional layer 212 and the domain-independent channel attention extractor 190.

[0127] The convolutional layer 212 generates a preliminary attention map from the feature map output by the convolutional layer 211 and outputs it to the integrating unit 202. The preliminary attention map output at this time has, as an example, K channels and W×H pixels. In FIG. 7, W×H×K output from the convolutional layer 212 represents the data size (dimension) of the preliminary attention map. The domain-independent channel attention extractor 190 outputs channel attention related to the feature output from the convolutional layer 212. The channel attention has the meaning of a weighting coefficient for each channel used when generating the attention map. In FIG. 7, 1×1×K output from the domain-independent channel attention extractor 190 represents the data size (dimension) of the channel attention.

[0128] The integrator 202 generates an attention map from the channel attention output from the domain-independent channel attention extractor 190 and the preliminary attention map output from the convolutional layer 212. For example, the integrator 202 generates an attention map for each channel of the preliminary attention map by weighted linear sum using each coefficient included in the channel attention. This results in the generation of the attention map 182 with one channel.

[0129] (Domain-independent channel attention extractor) Fig. 8 is a diagram illustrating a detailed configuration example of the domain-independent channel attention extractor 190 in Fig. 7. As shown in Fig. 8, the domain-independent channel attention extractor 190 includes a CNN 191, a domain classification unit 192, a conversion unit 193, and an integration unit 194.

[0130] In FIG. 8, the data size of the feature map (the "further feature" described above) subjected to pre-processing by the convolutional layer 211 is represented as W×H×C. This feature map is input to the CNN 191, which is a prediction model in the domain-independent channel attention extractor 190. The CNN 191 extracts features from the feature map subjected to pre-processing to generate a feature map (second feature) with K channels and W×H pixels, and supplies the feature map to the domain classification unit 192. Note that the feature map 181, which is the first feature extracted by the feature extraction unit 22, may be supplied directly to the domain classification unit without going through the CNN 191.

[0131] In this way, the CNN 191 functions as a second feature extraction unit that extracts second features from the above-mentioned first features or features obtained from the first features. The domain classification unit 192 performs domain classification on the feature map output from the CNN 191 or the feature map output from the feature extraction unit 22.

[0132] Here, the classification (prediction) process performed by the domain classification unit 192 includes the following two processes.

[0133] (Classification process 1) Domain classification (prediction) process for the entire input feature map (Classification process 2) Domain classification (prediction) process for each channel in the input feature map In (Classification Process 1), the domain classification unit 192 calculates the probability that the domain to which the entire feature map belongs is a source domain (e.g., a CG image) and the probability that it is a target domain (e.g., a real-life image). At this time, the domain classification unit 192 calculates the classification result by performing global average pooling processing and full connection processing on a feature map (data size W×H×K) with K channels. In FIG. 8, the classification result of the entire feature map by the domain classification unit 192 is represented as 1×1×2.

[0134] In (classification process 2), the domain classification unit 192 calculates the probability that each channel of the feature map output from the CNN 191 is the source domain and the probability that it is the target domain. In FIG. 8, the classification result by the domain classification unit 192 for each channel is expressed as 2×K. Here, the classification result by the domain classification unit 192 for each channel can be extracted as a weighting coefficient in the domain classification unit 192 (more specifically, the FC layer 192b described later). As described above, the attention map generation unit 23 includes the CNN 191 that extracts second features from first features or features obtained from the first features, and the domain classification unit 192 that performs domain classification based on the second features.

[0135] The conversion unit 193 converts the domain classification result for the entire feature map, which is classified (predicted) by the domain classification unit 192, and supplies the converted domain classification result to the integration unit 194.

[0136] The integration unit 194 calculates channel attention using the classification result for each channel by the domain classification unit 192 (in other words, the weighting coefficient in the domain classification unit 192) and the domain classification result after conversion by the conversion unit 193. Here, the data size of the channel attention is 1×1×K, as shown in Fig. 8. That is, the channel attention is expressed as one-dimensional (1×1) data for each of the K channels.

[0137] The conversion unit 193 A process of exchanging the confidence levels of each of the multiple domains in the domain classification result for the entire feature map; or A process for reducing the difference in confidence between each of a plurality of domains in the domain classification result for the entire feature map. The configuration can be such that the above is performed.

[0138] In addition, as an example, the data belonging to the target domain is live-action image data, and the conversion unit 193 treats the target domain to which the live-action images belong and the source domain to which the CG image data belongs as multiple domains, and generates a classification result after conversion.

[0139] By having the above-described configuration of the domain-independent channel attention extractor 190, the domain dependency of the attention map can be more effectively reduced.

[0140] If domain dependency is weakened, it is expected that the classification accuracy will improve by applying the generated attention map regardless of the domain the input data belongs to. For example, even if an attention map generated based on features extracted from a CG image is applied to a real-life image for classification processing, it will be possible to achieve favorable classification accuracy.

[0141] (Domain Classification Section 192) Fig. 9 is a diagram showing a detailed configuration example of the domain classification unit 192 shown in Fig. 8. As shown in Fig. 9, the domain classification unit 192 includes a GAP layer (global average pooling layer) 192a and an FC layer (fully connected layer) 192b. The GAP layer 192a performs global average pooling processing on an input feature map (data size: W × H × K) to generate post-GAP data (data size: 1 × 1 × K).

[0142] The FC layer 192b performs a full connection process on the data after GAP to calculate the domain classification (prediction) result for the entire input feature map. This process corresponds to the above-mentioned (Classification Process 1).

[0143] In addition, the domain classification (prediction) result for each channel can also be extracted from the FC layer 192b. This process corresponds to the above-mentioned (Classification Process 2). As an example, the domain classification (prediction) result for each channel can be extracted as a weighting coefficient in the FC layer by inputting obvious data (data whose values ​​are all 1) to the FC layer.

[0144] In this way, the domain classification unit 192 includes a global average pooling layer 192a that receives the second feature as input, and a fully connected layer 192b that receives the output of the global average pooling layer 192a as input, and the domain-independent channel attention extractor 190 is configured to use the weighting coefficients of the fully connected layer 192b as the weighting coefficients of the domain classification unit 192.

[0145] According to the domain-independent channel attention extractor 190 including the domain classification unit 192 configured as above, it is possible to more suitably reduce the domain dependency of the attention map.

[0146] (Flow of identification process by information processing device 100) Next, the flow of the identification process performed by the information processing device 100 will be described with reference to FIG.

[0147] (Step S111) In step S111, the acquisition unit 21 acquires the target domain data or the source domain data.

[0148] (Step S112) In step S112, the feature extraction unit 22 extracts a first feature from the data acquired in step S111.

[0149] (Step S113) In step S113, the attention map generating unit 23 generates an attention map.

[0150] (Step S114) In step S114, the classification unit 24 performs classification processing with reference to the attention map. The classification processing by the classification unit 24 has been described above, so a description thereof will be omitted here.

[0151] (Attention map generation process flow) Next, the flow of the attention map generation process (step S113 in FIG. 10) by the attention map generation unit 23 will be described with reference to FIG.

[0152] (Step S121) In step S121, the convolutional layer 211 extracts an attention map feature from the first feature. The extracted attention map feature is provided to the convolutional layer 212.

[0153] (Step S122) In step S122, the convolutional layer 212 generates (predicts) a preliminary attention map (data size: W×H×K) from the attention map features extracted in step S121.

[0154] (Step S123) In step S123, the domain-independent channel attention extractor 190 generates and outputs the channel attention.

[0155] (Steps S124 and S125) In steps S124 and S125, the integrating unit 202 generates an attention map by integrating the preliminary attention map and the channel attention. For example, the integrating unit 202 generates the attention map by multiplying the preliminary attention map by the channel attention.

[0156] (Channel attention generation process flow) Next, with reference to FIG. 12, the flow of the channel attention generation process (step S123 in FIG. 11) by the domain-independent channel attention extractor 190 will be described.

[0157] (Step S131) In step S131, the CNN 191 extracts a second feature amount. The extraction process of the second feature amount by the CNN 191 has been described above, and therefore will not be described here.

[0158] (Step S132) In step S132, the domain classification unit 192 performs domain classification (prediction).

[0159] (Step S133) In step S133, the domain classification unit 192 outputs the weighting coefficients in the domain classification unit 192. The weighting coefficients have been described above, so a description thereof will be omitted here.

[0160] (Step S134) In step S134, the conversion unit 193 converts the domain classification result for the entire feature map obtained by the domain classification unit 192. The conversion process has been described above, and therefore will not be described here.

[0161] (Step S135) In step S135, the integration unit 194 calculates the channel attention using the domain classification (prediction) result after conversion in step S134 and the weighting coefficient output in step S133.

[0162] According to the information processing device 100 that executes the above-described processing, the domain dependency of the attention map can be more suitably reduced, and therefore classification processing relating to the target domain can be suitably performed.

[0163] Exemplary Embodiment 3 Next, a third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0164] Fig. 13 is a block diagram showing an example of the configuration of an information processing device according to exemplary embodiment 3. The information processing device 100 shown in Fig. 13 is configured by adding a label data generation unit 300 to the information processing device 100 according to exemplary embodiment 2 shown in Fig. 6. The other configurations are the same as those of the information processing device 100 in Fig. 6.

[0165] The label data generation unit 300 is a functional block that generates a CG image with an attention map as label data of the attention map required for training the attention map generation unit 23. Here, an example will be described in which the source domain is a CG image, the target domain is a real-life image, and the information processing device 100 identifies the model of an automobile shown in an image acquired by the acquisition unit 21.

[0166] The label data generation unit 300 generates a CG image with an attention map, which is a CG image with an attention map attached, from CG models related to multiple car models. Figure 14 is a diagram explaining the generation of a CG image with an attention map from a CG model.

[0167] 14, a CGM is shown as a CG model of a specific car model. The CGM is a three-dimensional model of the car model and has the same shape as an actual car. The label data generation unit 300 stores multiple CG models corresponding to multiple car models.

[0168] For example, the user of the information processing device 100 specifies a portion of interest in the CGM. As an example, the portion of interest is specified by selecting a part of an automobile, such as a headlight or a front grille.

[0169] For example, a point of interest in the CGM is specified by an operation such as selecting a desired part on a CG model of an automobile displayed on a computer display. In this example, a front window part 401 and a headlight and front grill part 402 are specified as points of interest in the CGM of the automobile shape.

[0170] Note that there are multiple CG models depending on the type of automobile, and a point of interest is specified for each of the multiple CG models. When a point of interest is specified by the user, information identifying each CG model (e.g., model number) and information identifying the part on the CG model corresponding to the point of interest (e.g., part number) are stored as data.

[0171] The label data generation unit 300 generates multiple CG images by rendering a CG model at multiple camera angles. For example, the label data generation unit 300 generates multiple CG images by rotating the CGM in Fig. 14 in multiple directions. Information specifying the direction and amount of rotation of the CGM is stored in advance as, for example, a camera angle pattern.

[0172] The label data generator 300 then generates an attention map by identifying points of interest in multiple CG images corresponding to multiple camera angles. The attention map thus generated serves as the label data for the attention map.

[0173] That is, the data belonging to the source domain is CG image data, and the information processing device 100 is provided with a label data generation unit 300 that generates label data for the attention map by rendering a CG model at a predetermined camera angle to generate a CG image and outputting a designated part that is a pre-designated part of the CG model as an attention map in the CG image.

[0174] (Configuration of label data generation unit 300) Fig. 15 is a block diagram showing a detailed example configuration of the label data generation unit 300 in Fig. 13. As shown in Fig. 15, the label data generation unit 300 has a CG model DB 311, a specified parts DB 312, a camera angle pattern 313, a CG attention map DB 314, and a CG image DB 315. These functional blocks are configured as a database and are stored in, for example, a storage unit configured by a semiconductor memory device or the like. The storage unit may be provided inside the information processing device 100, or may be connected to the information processing device 100 as needed via, for example, the communication unit 162.

[0175] The CG model DB 311 stores a plurality of CG models corresponding to a plurality of vehicle models. The specified parts DB 312 stores information (e.g., a pair of a model number and a part number) that identifies a part designated as a focus point in each CG model (hereinafter referred to as a specified part, as appropriate). The camera angle pattern DB 313 stores information (hereinafter referred to as a camera angle pattern, as appropriate) that indicates in what direction and by how much the CG model is to be rotated when the CG model is rendered.

[0176] The CG image DB 315 stores CG images obtained by rendering CG models. The CG image DB 315 stores CG images rendered at multiple camera angles for each of multiple CG models. The CG attention map DB 314 stores attention maps corresponding to each CG image obtained by rendering CG models.

[0177] As shown in FIG. 15, the label data generating unit 300 includes a model reading unit 331, a specified parts reading unit 332, and an attention-attached rendering processing unit 333.

[0178] The model reading unit 331 reads out a CG model stored in the CG model DB 311. For example, a CG model corresponding to the vehicle model A01 and having the model number A01 is read out.

[0179] The designated parts readout unit 332 reads out designated parts stored in the designated parts DB. For example, if the designated parts for the vehicle model A01 read out by the model readout unit 331 are headlights and a front grille, then the part number 001 for the headlights and the part number 002 for the front grille are read out.

[0180] The attention-attached rendering processor 333 identifies the camera angle based on the information read from the camera angle pattern DB and creates a CG image by rendering a CG model. The attention-attached rendering processor 333 also generates an attention map for the created CG image.

[0181] There are various methods for generating attention maps when rendering CG images.

[0182] As an example, the attention-based rendering processor 333 generates an attention map by coloring parts designated as attention points in a CG model with a predetermined color and calculating the positions of pixels of that color in the rendered CG image. This method will be referred to as the first method of attention map generation.

[0183] As an example, the attention-based rendering processor 333 generates an attention map by calculating the intersection position between the surface of a part designated as a focus point in the rendered CG image and the light beam from the camera. This method will be referred to as the second method of attention map generation.

[0184] (Configuration Example 1 of Attention-attached Rendering Processor 333) 16 is a block diagram showing a detailed example of the configuration of the attention-attached rendering processor 333 in FIG. 15, and shows an example of the configuration corresponding to the first method described above. In this example, the attention-attached rendering processor 333 has a designated parts color processor 351-1, a non-designated parts color processor 351-2, a rendering processor 352, and a pixel value threshold processor 353.

[0185] The specified parts color processing unit 351-1 colors the specified parts read by the specified parts reading unit 332 in the CG model read by the model reading unit 331 in a preset first color. The first color is, for example, a color that is brighter than a predetermined brightness. The non-specified parts color processing unit 351-2 colors the parts other than the specified parts read by the specified parts reading unit 332 in the CG model read by the model reading unit 331 in a preset second color. The second color is, for example, a color that is darker than a predetermined brightness.

[0186] The rendering processing unit 352 reads out a camera angle pattern from the camera angle pattern DB 313. Then, the rendering processing unit 352 applies the read-out camera angle pattern to render the CG model colored by the designated parts color processing unit 351-1 and the non-designated parts color processing unit 351-2.

[0187] The pixel value threshold processing unit 353 generates and outputs an attention map by identifying pixels having pixel values ​​equal to or greater than a threshold from the CG image obtained through processing by the rendering processing unit 352. This results in an attention map corresponding to the positions of points of interest in the image of the car captured at that camera angle.

[0188] In this way, the attention rendering processing unit 333 shown in Figure 16 colors the specified part of the CG model with a predetermined color and identifies pixels of the predetermined color from the CG image, thereby outputting an attention map within the CG image.

[0189] The rendering processing unit 352 renders the CG model and stores the generated CG image in the CG image DB 315. The pixel value threshold processing unit 353 also stores the generated attention map in the CG attention map DB 314.

[0190] (Configuration Example 2 of Attention-attached Rendering Processor 333) 17 is a block diagram showing another detailed configuration example of the attention-attached rendering processor 333 of FIG. 15, which corresponds to the second method described above. In this example, the attention-attached rendering processor 333 has a rendering processor 371, a camera-origin ray processor 372, a pixel-by-pixel collision determination unit 373, and a foreground determination unit 374.

[0191] The rendering processing unit 371 reads out a camera angle pattern from the camera angle pattern DB 313. Then, the rendering processing unit 371 applies the read-out camera angle pattern to render the CG model read out by the model reading unit 331. The rendering processing unit 371 also stores the CG image generated by rendering the CG model in the CG image DB 315.

[0192] The camera origin ray processing unit 372 calculates rays connecting each point in the field of view with the camera during rendering.

[0193] The pixel-by-pixel collision determination unit 373 determines whether the ray calculated by the camera-origin ray processing unit 372 collides with each part of the CG model, thereby calculating the depth value of each part of the CG model. At this time, the location where the ray collides with the surface of the part is identified, and this location becomes the intersection position. In addition, a value indicating how far the intersection position is from the camera becomes the depth value.

[0194] The foreground determination unit 374 determines whether the part having the depth value closest to the camera from the calculated depth values ​​is the designated part, and generates and outputs an attention map by identifying the area where the part closest to the camera is the designated part. This results in an attention map corresponding to the positions of attention points in the image of the car captured at that camera angle. The foreground determination unit 374 also stores the generated attention map in the CG attention map DB 314.

[0195] In this way, the attention-attached rendering processing unit 333 shown in FIG. 17 outputs an attention map within the CG image based on the intersection position between the ray originating from the camera in rendering and the designated part of the CG model.

[0196] (Flow of image generation process with attention map) Next, an example of the processing for generating an image with an attention map by the label data generating unit 300 shown in Fig. 15 will be described. Fig. 18 is a flowchart illustrating the flow of the processing for generating an image with an attention map.

[0197] In step S301, the model reading unit 331 reads a CG model from the CG model DB 311. At this time, for example, a CG model corresponding to the vehicle model A01 and having the model number A01 is read.

[0198] In step S302, the specified parts reader 332 reads out the specified parts stored in the specified parts DB. For example, if the specified parts for the vehicle model A01 read out by the model reader 331 are a headlight and a front grille, then the part number 001 for the headlight and the part number 002 for the front grille are read out.

[0199] In step S303, the attention-based rendering processor 333 executes rendering processing. This processing creates a CG image by rendering a CG model according to a camera angle pattern read from the camera angle pattern DB. The attention-based rendering processor 333 also generates an attention map for the created CG image.

[0200] As described above, the attention-attached rendering processing unit 333 configured as described above with reference to Fig. 16 is configured to correspond to the first method, and the attention-attached rendering processing unit 333 configured as described above with reference to Fig. 17 is configured to correspond to the second method. Below, the rendering processing of step S303 will be explained separately as processing corresponding to the first method and processing corresponding to the second method.

[0201] (Rendering process A flow) Here, a rendering process A, which is an example of the rendering process of step S303 in Fig. 18, will be described with reference to the flowchart in Fig. 19. This process corresponds to the first method described above, and is executed by the attention-attached rendering processing unit 333 configured as described above with reference to Fig. 16.

[0202] In step S321, the designated parts color processing unit 351-1 colors the designated parts read in step S302 in the CG model read in step S301 with a preset first color. The first color is, for example, a color brighter than a predetermined brightness.

[0203] In step S322, the non-designated parts color processing unit 351-2 colors, in a preset second color, the parts of the CG model read by the model read unit 331, other than the designated parts read by the designated parts read unit 332. The second color is, for example, a color darker than a predetermined brightness.

[0204] In step S323, the rendering processing unit 352 renders the CG model colored in steps S321 and S322 in accordance with the camera angle pattern read out from the camera angle pattern DB 313.

[0205] In step S324, the pixel value threshold processing unit 353 generates an attention map by identifying pixels having pixel values ​​equal to or greater than the threshold from the CG image obtained as a result of the processing in step S323. This generates an attention map corresponding to the positions of attention points in the image of the car captured at that camera angle.

[0206] In this way, the rendering process A corresponding to the first method described above is executed. In this way, the CG model can be rendered at various camera angles, and CG images corresponding to camera angles that may exist in live-action images can be generated. Furthermore, once a part is designated as a focus point, its position can be accurately identified in CG images taken at various camera angles, and an attention map can be generated.

[0207] (Rendering process B flow) Next, a rendering process B, which is another example of the rendering process in step S303 in Fig. 18, will be described with reference to the flowchart in Fig. 20. This process corresponds to the second method described above, and is executed by the attention-attached rendering processing unit 333 configured as described above with reference to Fig. 17.

[0208] In step S341, the rendering processing unit 371 renders the CG model read out in step S301 in accordance with the camera angle pattern read out from the camera angle pattern DB 313.

[0209] In step S342, the camera origin ray processing unit 372 calculates rays connecting each point in the field of view at the time of rendering with the camera.

[0210] In step S343, pixel-by-pixel collision determination unit 373 determines whether the ray calculated in step S342 will collide with each part of the CG model, thereby calculating the depth value of each part of the CG model.

[0211] In step S344, the foreground determination unit 374 determines whether the part having the depth value closest to the camera is the designated part based on the calculated depth values, and identifies the area where the part closest to the camera is the designated part. This generates an attention map corresponding to the positions of attention points in the image of the car captured at that camera angle.

[0212] In this way, rendering process B corresponding to the second method described above is executed. In this way, the CG model can be rendered at various camera angles, and CG images corresponding to possible camera angles in live-action images can be generated. Furthermore, once a part is designated as a focus point, its position can be accurately identified in CG images from various camera angles, and an attention map can be generated, without the need to color the designated part.

[0213] [Software implementation example] Some or all of the functions of the information processing device 20 and the information processing device 100 may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0214] In the latter case, the information processing device 20 and the information processing device 100 are realized by, for example, a computer that executes instructions of a program that is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG.

[0215] The computer C includes at least one processor C101 and at least one memory C102. The memory C102 stores a program P for causing the computer C to operate as the information processing device 20 or the information processing device 100. In the computer C, the processor C101 reads and executes the program P from the memory C102, thereby realizing each function of the information processing device 20 or the information processing device 100.

[0216] The processor C101 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C102 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0217] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0218] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0219] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0220] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

[0221] (Appendix 1) an acquisition means for acquiring data belonging to a target domain; a first feature extraction means for extracting a first feature from the data; an attention map generating means for generating an attention map from the first feature amount, the attention map generating means being configured to reduce domain dependency of the generated attention map; a classification means for performing a classification process on the data based on the first feature amount and the attention map; An information processing device comprising:

[0222] (Appendix 2) The attention map generating means a second feature extraction means for extracting a second feature from the first feature or a feature obtained from the first feature; a classification means for performing domain classification based on the second feature amount; a channel attention calculation means for calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classification means and a classification result by the classification means; an attention map calculation means for calculating the attention map using the first feature amount or a feature amount obtained from the first feature amount and the channel attention; 2. The information processing device according to claim 1, comprising:

[0223] (Appendix 3) The channel attention calculation means a conversion means for converting a classification result by the classification means, The channel attention is calculated using the weighting coefficient of the classification means and the classification result after conversion by the conversion means. 3. The information processing device according to claim 2.

[0224] (Appendix 4) The conversion means A process of exchanging the confidence levels of each of the multiple domains in the classification result with each other; or A process for reducing the difference in the confidence levels of the multiple domains in the classification results. Generate the transformed classification result by 4. The information processing device according to claim 3.

[0225] (Appendix 5) the data belonging to the target domain is live-action image data, The conversion means sets the target domain to which the live-action image data belongs and the source domain to which the CG image data belongs as the plurality of domains, and generates the classification result after the conversion. 5. The information processing device according to claim 4.

[0226] (Appendix 6) The classification means a global average pooling layer that receives the second feature as an input; a fully connected layer that receives the output of the global average pooling layer as input; It is equipped with The channel attention calculation means The weight coefficients of the fully connected layer are used as the weight coefficients of the classification means. 6. An information processing device according to any one of appendixes 2 to 5.

[0227] (Appendix 7) Obtaining data belonging to the target domain; extracting a first feature from the data; generating an attention map from the first feature amount such that domain dependency of the generated attention map is reduced; performing a classification process for the data based on the first feature amount and the attention map; An information processing method including:

[0228] (Appendix 8) Computer, an acquisition means for acquiring data belonging to a target domain; a first feature extraction means for extracting a first feature from the data; an attention map generating means for generating an attention map from the first feature amount, the attention map generating means being configured to reduce domain dependency of the generated attention map; and causing the information processing device to function as an information processing device having a discrimination means for performing a discrimination process on the data based on the first feature amount and the attention map. program.

[0229] (Appendix 9) an acquisition means for acquiring data belonging to a source domain and label data of an attention map associated with the data; a feature extraction means for extracting a first feature from the data; a learning means for learning, using the label data, an attention map generating means for generating an attention map from the feature amount of the data and the first feature amount, the attention map generating means being configured to reduce domain dependency of the attention map to be generated; An information processing device comprising:

[0230] (Appendix 10) The attention map generating means a second feature extraction means for extracting a second feature from the first feature; a classification means for performing domain classification based on the second feature amount; a channel attention calculation means for calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classification means and a classification result by the classification means; Equipped with The learning means, in the learning process of the attention map generation means, without updating the weighting coefficient of the second feature amount extraction means, updating the weighting coefficients of the classifier; 10. The information processing device according to claim 9.

[0231] (Appendix 11) the data belonging to the source domain is CG image data, CG models are rendered at a specific camera angle to generate CG images. further comprising label data generating means for generating label data of the attention map by outputting a designated portion, which is a portion designated in advance in the CG model, as an attention map in the CG image; The learning means causes the attention map generating means to learn using the label data generated by the label data generating means. 11. The information processing device according to claim 9 or 10.

[0232] (Appendix 12) The label data generating means coloring the designated portion of the CG model with a predetermined color; By identifying pixels of the predetermined color from the CG image, an attention map in the CG image is output. 12. The information processing device according to claim 11.

[0233] (Appendix 13) The label data generating means An attention map in the CG image is output based on an intersection position between a ray originating from a camera in the rendering and the specified portion of the CG model. 12. The information processing device according to claim 11.

[0234] (Appendix 14) Obtaining data belonging to a source domain and label data of an attention map associated with the data; extracting a first feature from the data; An attention map generating means that generates an attention map from the feature amount of the data and the first feature amount, the attention map generating means being configured to reduce domain dependency of the attention map to be generated, is trained using the label data. An information processing method including:

[0235] (Appendix 15) Computer, an acquisition means for acquiring data belonging to a source domain and label data of an attention map associated with the data; a feature extraction means for extracting a first feature from the data; The information processing device functions as an information processing apparatus including: attention map generation means for generating an attention map from the feature amount of the data and the first feature amount, the attention map generation means being configured to reduce domain dependency of the generated attention map; and learning means for learning the attention map using the label data. program.

[0236] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows.

[0237] at least one processor, the processor comprising: retrieving data belonging to a target domain; extracting a first feature amount from the data; an attention map generation process for generating an attention map from the first feature amount, the attention map generation process being executed so as to reduce domain dependency of the generated attention map; and performing a process of identifying the data based on the first feature amount and the attention map. Information processing device.

[0238] at least one processor, the processor comprising: A process of acquiring data belonging to a source domain and label data of an attention map associated with the data; extracting a first feature amount from the data; and performing a process of learning the attention map generation process using the label data, the attention map generation process being executed to generate an attention map from the feature amount of the data and the first feature amount so that the domain dependency of the generated attention map is reduced. Information processing device.

[0239] These information processing devices may further include a memory that stores a program for causing the processor to execute the process of acquiring label data, the process of extracting first feature amounts, and the process of generating an attention map. The program may also be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0240] 20 Information processing equipment 21 Acquisition Department 22 Feature extraction unit 23 Attention map generation unit 24 Identification unit 121A Data acquisition unit 121B Data acquisition section 122A Feature Extraction Unit 122B Feature Extraction Unit 190 Domain-independent channel attention extractor 192 Domain Classification Department 192a GAP layer 192b FC layer 300 Label data generation unit 331 Model Readout Unit 332 Specified parts reading unit 333 Attention-based rendering processor

Claims

1. an acquisition means for acquiring data belonging to a target domain; a first feature extraction means for extracting a first feature from the data; an attention map generating means for generating an attention map from the first feature amount, the attention map generating means being configured to reduce domain dependency of the generated attention map; a classification means for performing a classification process on the data based on the first feature amount and the attention map; Equipped with The attention map generating means a second feature extraction means for extracting a second feature from the first feature or a feature obtained from the first feature; a classification means for performing domain classification based on the second feature amount; a channel attention calculation means for calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classification means and a classification result by the classification means; an attention map calculation means for calculating the attention map using the first feature amount or a feature amount obtained from the first feature amount and the channel attention; An information processing device comprising:

2. The channel attention calculation means a conversion means for converting a classification result by the classification means, The channel attention is calculated using the weighting coefficient of the classification means and the classification result after conversion by the conversion means. The information processing device according to claim 1 .

3. The conversion means A process of exchanging the confidence levels of each of the multiple domains in the classification result with each other; or A process for reducing the difference in the confidence levels of the multiple domains in the classification results. Generate the transformed classification result by The information processing device according to claim 2 .

4. the data belonging to the target domain is live-action image data, The conversion means sets the target domain to which the live-action image data belongs and the source domain to which the CG image data belongs as the plurality of domains, and generates the classification result after the conversion. The information processing device according to claim 3 .

5. The classification means a global average pooling layer that receives the second feature as an input; a fully connected layer that receives the output of the global average pooling layer as input; It is equipped with The channel attention calculation means The weight coefficients of the fully connected layer are used as the weight coefficients of the classification means. The information processing device according to claim 1 .

6. A computer comprising: Obtaining data belonging to the target domain; extracting a first feature from the data; generating an attention map from the first feature amount such that domain dependency of the generated attention map is reduced; performing a classification process for the data based on the first feature amount and the attention map; Including, In generating the attention map, extracting a second feature amount from the first feature amount or a feature amount obtained from the first feature amount; performing domain classification based on the second feature amount; calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient in the domain classification and a classification result of the domain classification; calculating the attention map using the first feature amount or a feature amount obtained from the first feature amount and the channel attention; An information processing method including:

7. Computer, an acquisition means for acquiring data belonging to a target domain; a first feature extraction means for extracting a first feature from the data; an attention map generating means for generating an attention map from the first feature amount, the attention map generating means being configured to reduce domain dependency of the generated attention map; a classification means for performing a classification process on the data based on the first feature amount and the attention map, In the attention map generating means, a second feature extraction means for extracting a second feature from the first feature or a feature obtained from the first feature; a classification means for performing domain classification based on the second feature amount; a channel attention calculation means for calculating a channel attention, which is a weighting coefficient for each channel related to the second feature amount, using the weighting coefficient of the classification means and a classification result by the classification means; and an attention map calculation means for calculating the attention map using the first feature amount or a feature amount obtained from the first feature amount and the channel attention. program.

Images