Learning method and recognition device

Abstract

To improve recognition accuracy when a recognition apparatus is trained using real data actually acquired from a class to be recognized and virtual data virtually generated as learning data.SOLUTION: A learning method comprises: training a feature extraction unit and a class identification unit so that a class identified by the class identification unit matches a first class based on a feature extracted by the feature extraction unit from actual data of the first class (S1); inputting first virtual data of the first class and first real data of the first class to the feature extraction unit, and extracting a first virtual data feature that is the feature of the first virtual data and a first real data feature that is the feature of the first real data (S2, S3); and training the feature extraction unit so that a data discriminator cannot classify the first virtual data feature and the first real data feature (S4 to S6), wherein the data discriminator classifies whether it is the feature of the actual data or the feature of the virtual data.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a learning method and a recognition device. [Background technology]

[0002] Patent Document 1 describes a recognition system that includes a feature extraction means that extracts features of an input image, a class identification means that identifies the class of the input image based on the features extracted by the feature extraction means, and a learning means that trains the feature extraction means based on training images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-117565 Summary of the Invention [Problem to be solved by the invention]

[0004] When training a recognition device such as the one described above, it may be costly to actually collect data on the class of objects to be recognized in various environments. Therefore, by virtually generating data on the class of objects to be recognized based on real data and using the data as training data, the cost required for training the object of recognition can be reduced. For example, when images of the class of objects to be recognized are used as input data, computer graphics (CG) images of the class of objects to be recognized are generated, and the CG images are used as training data to train the recognition device. In the following description, data actually collected from the class of objects to be recognized will be referred to as "real data," and data on the class of objects to be recognized virtually generated based on the real data will be referred to as "virtual data."

[0005] However, because virtual data is generated under certain assumptions and constraints, it differs from real data in strict terms. The feature distribution extracted from such virtual data will deviate from the feature distribution extracted from real data. Therefore, when a recognition device is trained using virtual data, the feature extraction means may be trained so that the inter-class distances of the feature distribution extracted when real data is input are uneven in the feature space. As a result, when the probability distribution of each class (i.e., the probability that each class is the correct class) is calculated in class identification, the probability of classes that are not the correct class may be erroneously inferred to be high, resulting in a decrease in recognition accuracy. The present invention has been made in consideration of the above-mentioned problems, and aims to improve the recognition accuracy when training a recognition device using real data actually obtained from a class to be recognized and virtually generated virtual data as training data. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a training method for a recognition device including a feature extraction unit that extracts features of input data and a class identification unit that identifies a class of the input data based on the features extracted by the feature extraction unit. The training method includes training the feature extraction unit and the class identification unit so that a class identified by the class identification unit based on the features extracted by the feature extraction unit from actual data of a first class matches the first class, inputting first virtual data that is virtual data of the first class generated based on the actual data of the first class and first actual data that is the actual data of the first class to the feature extraction unit, extracting first virtual data features that are features of the first virtual data and first actual data features that are features of the first actual data, and training the feature extraction unit so that a data classifier that classifies features as actual data or virtual data cannot classify the first virtual data features into the first actual data features. According to another aspect of the present invention, there is provided a recognition device comprising a feature extraction unit trained by the above-described training method and a class identification unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the recognition accuracy when training a recognition device using real data actually acquired from a class to be recognized and virtually generated virtual data as training data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram illustrating an example of a hardware configuration of a recognition device according to an embodiment of the present invention. [Figure 2] 1 is a block diagram illustrating an example of a functional configuration of a recognition device according to an embodiment of the present invention. [Figure 3] 10(a) to 10(d) are schematic diagrams of a feature space in which the feature values ​​output from the feature extracting unit are distributed. [Figure 4] 10A and 10B are schematic diagrams showing the distribution of feature amounts output from the feature amount extraction unit when real data and virtual data are input, and the discrimination boundary of the data discrimination unit. [Figure 5] 10 is a flowchart illustrating an example of a learning method for a feature extraction unit of a recognition device according to an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an example of a learning method for a data identification unit of a recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments of the present invention shown below are merely examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of the components to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0010] As an example of an embodiment of the present invention, an example of a recognition device will be described that accepts an image as input data, extracts features of the input image of a specific class (e.g., a person), and recognizes the class of the input image based on the extracted features. The recognition device may perform, for example, face recognition or person identification (Re-Identification: Re-ID). However, the present invention is not limited to this and can also be applied to a recognition device that accepts a voice signal as input data and recognizes the class of the input voice. The present invention can also be applied to a recognition device that accepts a range image as input data and recognizes the class of a point cloud included in the range image.

[0011] 1 is a schematic diagram showing an example of the hardware configuration of a recognition device 1 according to an embodiment of the present invention. The recognition device 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, an output unit 6, and an operation input unit 7. The image capturing unit 2 is a surveillance camera installed for the purpose of monitoring a predetermined area, and is attached in a position where it can capture images of people staying within the area. The images captured by the image capturing unit 2 are transmitted to the image processing unit 5 via the communication unit 3.

[0012] The communication unit 3 transmits and receives data between the imaging unit 2, the image processing unit 5, the output unit 6, and the operation input unit 7. A LAN (Local Area Network) or a public line such as the Internet can be used. The storage unit 4 is configured by a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system, and various data. The image processing unit 5 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and transmits the results of image processing performed on the image captured by the imaging unit 2 to the output unit 6 via the communication unit 3.

[0013] The output unit 6 is a display, a projector, a printer, a removable drive, a USB (Universal Serial Bus) interface, a network interface, or the like that outputs various information generated by the recognition device 1. The operation input unit 7 is a mouse, keyboard, etc. that is operated by the user to accept inputs specifying templates and search ranges.

[0014] 2 is a block diagram of an example of the functional configuration of a recognition device 1 according to an embodiment of the present invention. The recognition device 1 is a device that acquires data of a class to be recognized (hereinafter, may be referred to as a "class to be recognized") as input data, extracts features of the input data, and recognizes the class of the input data based on the extracted features. The recognition device 1 includes a real data acquisition unit 10, a virtual data generation unit 11, a feature extraction model storage unit 12, a feature extraction unit 13, a template feature storage unit 14, a class identification unit 15, a data identification unit 16, a data identification model storage unit 17, a learning unit 18, and a classification result output unit 19. The data identification unit 16 is an example of a "data classifier" as defined in the claims.

[0015] The photographing unit 2 or the memory unit 4 in FIG. 1 functions as an actual data acquisition unit 10, the memory unit 4 functions as a feature extraction model memory unit 12, a template feature memory unit 14, and a data classification model memory unit 17, the image processing unit 5 functions as a virtual data generation unit 11, a feature extraction unit 13, a class classification unit 15, a data classification unit 16, and a learning unit 18, and the output unit 6 functions as a classification result output unit 19.

[0016] When training the recognition device 1, the real data acquisition unit 10 acquires real data DR that is actually collected from a class that is a learning target (hereinafter, may be referred to as a "learning target class") of the recognition device 1. When the recognition device 1 recognizes a class of input data, the real data acquisition unit 10 acquires real data DR that is actually collected from the recognition target class.

[0017] For example, when an image is used as input data, the actual data acquisition unit 10 may acquire, as the actual data DR, an image obtained by actually photographing the target class with the photographing unit 2. For example, when audio data is used as input data, the actual data acquisition unit 10 may acquire, as the actual data DR, an audio signal obtained by actually collecting audio generated from the target class with an audio input device (microphone) not shown. Furthermore, when a distance image is used as input data, for example, the actual data acquisition unit 10 may acquire, as the actual data DR, point cloud data obtained by actually measuring the target class with a distance measuring device not shown.

[0018] In addition, when acquiring actual data DR for learning to be used for training the feature extraction unit 13 and the class identification unit 15, the actual data acquisition unit 10 may acquire as the actual data DR data of a known learning target class that has been prepared in advance as learning data and stored in the memory unit 4. The actual data acquisition unit 10 inputs the acquired actual data DR to the feature extraction unit 13. The virtual data generating unit 11 virtually generates virtual data DV of the learning target class based on the real data DR of the learning target class acquired by the real data acquiring unit 10. For example, when an image is used as input data, CG processing may be performed on the actual data DR to generate, as virtual data DV, a CG image in which the class appears in an orientation different from the original orientation of the class appearing in the actual data DR. For example, when an image such as a face image taken from the front is acquired as the actual data DR, a CG image facing sideways or a CG image facing downward may be generated as virtual data DV.

[0019] As a specific example, the virtual data generation unit 11 estimates a three-dimensional model of a class depicted in the real data DR from the real data DR and extracts texture, which is pixel information (brightness, saturation, hue) of each coordinate on the surface of the class on the real data DR. The virtual data generation unit 11 changes the orientation of the three-dimensional model to a desired orientation and adds texture so that pixel information at the same coordinate on the surface of the three-dimensional model does not change before and after the orientation change, thereby generating virtual data DV.

[0020] Furthermore, when audio data is used as input data, the speech rate (pitch) of the audio data of the real data DR may be changed or external noise may be added to generate the virtual data DV. When a distance image is used as input data, the virtual data DV may be generated by changing the orientation of the point cloud of the distance image of the real data DR. The virtual data generation unit 11 inputs the acquired virtual data DV to the feature extraction unit 13.

[0021] The feature extraction model storage unit 12 stores a feature extraction model. For example, the feature extraction model may be modeled as a convolutional neural network (CNN) configured with a multi-layer network such as that used in deep learning. In this case, the feature extraction model has a network structure in which multiple layers, such as convolution layers, activation functions, and pooling layers, are connected in series, and the feature extraction model storage unit 12 stores information including the filter coefficients of the filters that make up the network and the network structure. These parameters of each layer, such as the filter coefficients, are updated through learning.

[0022] The feature extraction unit 13 extracts features of input data using a feature extraction model. That is, when real data DR is input from the real data acquisition unit 10, the feature extraction unit 13 extracts features of the real data DR, and when virtual data DV is input from the virtual data generation unit 11, the feature extraction unit 13 extracts features of the virtual data DV. The feature extraction unit 13 reads out the feature extraction model from the feature extraction model storage unit 12, inputs the input data to the feature extraction model, and outputs the features output by the feature extraction model to the class identification unit 15 and the data identification unit 16.

[0023] Template feature storage unit 14 stores template features representing features of the learning target class. For example, communication unit 3 may receive template features of the learning target class from an external device and store the received template features in template feature storage unit 14. Alternatively, template features generated by template feature generation means (not shown) from input data that is known to be data of the learning target class may be stored in template feature storage unit 14.

[0024] The class identification unit 15 reads out template features from the template feature storage unit 14, and also receives features extracted from the input data by the feature extraction unit 13. The class identification unit 15 compares the data input from the feature extraction unit 13 with the template features read out from the template feature storage unit 14, and outputs a probability distribution indicating the probability that the class of the input data is each of the learning target classes.

[0025] For example, the class identification unit 15 outputs a logit sequence a = (a1, a2, ...) (a = Wf) by multiplying the feature f by a matrix W = (w1, w2, ...) that represents the template feature of each learning target class Ck. Here, the logit ak represents ak = ln(p(x | Ck) p(Ck)). p(Ck) is the probability that the object to be identified belongs to the learning target class Ck, and p(x | Ck) is the probability density distribution that the feature vector x is observed when measuring an object that belongs to the learning target class Ck. Alternatively, the feature value f is multiplied by the matrix W to output a likelihood sequence p(x│.) = (p(x│C1), p(x│C2), ...) (p = Wf), and the logit sequence a is calculated from the logit definition formula above using the prepared prior distribution p(Ck). Then, we apply the softmac function p(Ck│x) = exp(ak) / (Σexp(aj)) to the logit sequence a to obtain the (posterior) probability sequence (i.e., probability distribution) p(.│x) = (p(C_1│x), p(C_2│x), …). The class identification unit 15 outputs the calculated probability distribution to the learning unit 18 and the identification result output unit 19. The identification result output unit 19 outputs the identification result of the class identification unit 15, that is, the recognition result of the class of the input data.

[0026] The learning unit 18 trains the feature extraction unit 13, the class identification unit 15, and the data identification unit 16. In training the feature extraction unit 13, the learning unit 18 calculates, for example, a loss function of the probability distribution output by the class identification unit 15, and trains a feature extraction model so as to lower the loss function. For example, the learning unit 18 calculates parameter update amounts for the feature extraction model to reduce the loss function using a gradient method or coordinate descent method with the loss function as an energy function, updates the feature extraction model by the update amounts, and then calculates and evaluates the loss function again, repeating this process until an iteration termination condition is met.

[0027] In training the class identification unit 15, the training unit 18 calculates, for example, a loss function of the probability distribution output by the class identification unit 15, and trains the template features stored in the template feature storage unit 14 so as to lower the loss function. For example, it calculates an update amount for the template features to reduce the loss function using a gradient method or coordinate descent method with the loss function as an energy function, updates the template features by the update amount, and then calculates and evaluates the loss function again, repeating this process until an iteration termination condition is satisfied.

[0028] In this case where the feature extraction unit 13 and the class identification unit 15 are trained simultaneously, if the learning unit 18 uses the real data DR and the virtual data DV without distinguishing between them, the inter-class distances of the feature distribution extracted by the feature extraction unit 13 when the real data DR is input may become uneven in the feature space. As a result, the class identification unit 15 may erroneously infer a high probability for a class different from the class of the input data, which may result in a decrease in the recognition accuracy of the recognition device. The reason for this will be explained with reference to Figures 3(a) to 3(c).

[0029] 3(a) to 3(d) are schematic diagrams of a feature space in which the features output from the feature extraction unit are distributed. The ranges DrA, DrB, DrC, and DrD enclosed by solid lines indicate the ranges in which the features of real data DR of different classes CA, CB, CC, and CD are distributed, respectively, and the ranges DvA, DvB, DvC, and DvD enclosed by dashed lines indicate the ranges in which the features of virtual data DV of classes CA to CD are distributed.

[0030] When the feature extraction unit 13 and the class identification unit 15 are trained using only the real data DR and its loss function, as shown in Figure 3(a), the training proceeds so as to reduce the intra-class variance (i.e., reduce the distribution) and increase the inter-class variance (increase the distribution distance between different classes) in the distribution range DrA to DrD (solid line) of the features of the real data DR extracted by the feature extraction unit 13. As a result, when training proceeds using only the real data DR and its loss function, the feature extraction unit 13 trains so as to create a feature space in which the inter-class distances of the distribution of the features of the real data DR are maintained uniform.

[0031] 3(b) shows the distribution ranges DvA to DvD (dashed lines) of the features of classes CA to CD output from the feature extraction unit 13 when virtual data DV is input. Although the virtual data DV is generated based on the real data DR, it is generated under certain assumptions and constraints and therefore differs from the real data DR in strict terms. For this reason, the distribution ranges DvA to DvD (dashed lines) of the features of the virtual data DV will deviate from the distribution ranges DrA to DrD (solid lines) of the features of the real data DR. In this case, the direction and extent of the deviation of the distribution will differ depending on the class.

[0032] Therefore, when the feature extraction unit 13 and the class identification unit 15 are trained using the virtual data DV and its loss function in addition to the real data DR, the distribution of the features extracted by the feature extraction unit 13 becomes as shown in Figure 3(c). In other words, the feature extraction unit 13 learns so that the feature space obtained by combining the distribution range of the features of the real data DR and the distribution range of the features of the virtual data DV (the combined range of distribution range DrA and distribution range DvA, the combined range of distribution range DrB and distribution range DvB, the combined range of distribution range DrC and distribution range DvC, and the combined range of distribution range DrD and distribution range DvD) maintains an equal inter-class distance.

[0033] As a result, the inter-class distances in the distribution ranges DrA to DrD (solid lines) of the feature amounts of the actual data DR become uneven. In this way, if the learning of the feature extraction unit 13 progresses so that the distribution range of the features of the real data DR becomes uneven, since the data input when class recognition is performed by the recognition device 1 is the real data DR, the class identification unit 15 may erroneously infer a high probability for a class different from the class of the input data, which may result in a decrease in the recognition accuracy of the recognition device.

[0034] For this reason, the recognition device 1 of the embodiment is equipped with a data identification unit 16 (see Figure 2) that identifies whether the features output from the feature extraction unit 13 are features of real data DR (hereinafter sometimes referred to as "real data features") or features of virtual data DV (hereinafter sometimes referred to as "virtual data features"). The data discrimination unit 16 reads out a data discrimination model stored in the data discrimination model storage unit 17. The data discrimination model is a discrimination model that discriminates whether the feature input from the feature extraction unit 13 is a real data feature or a virtual data feature. The data discrimination model outputs the probability that the feature input from the feature extraction unit 13 is a virtual data feature as a discrimination result. In the following description, the discrimination result of the data discrimination unit 16 will be referred to as the "virtual data probability." The data discrimination unit 16 outputs the virtual data probability to the learning unit 18.

[0035] The data identification model may be modeled, for example, as a CNN consisting of a multi-layer network such as that used in deep learning. In this case, the data identification model has a network structure in which multiple layers, such as convolutional layers, activation functions, and pooling layers, are connected in series, and the data identification model storage unit 17 stores information including the filter coefficients of the filters that make up the network and the network structure. These parameters of each layer, such as the filter coefficients, are updated through learning.

[0036] In training the recognition device 1, the training content of the training unit 18 differs between the real data DR and the virtual data DV. When only the real data DR is used as training data, the training unit 18 trains the feature extraction unit 13 and the class identification unit 15. When training the recognition device 1 using the virtual data DV as training data, the training unit 18 trains only the feature extraction unit 13 without training the class identification unit 15. This prevents the inter-class distances in the distribution ranges DrA to DrD (solid lines) of the features of the real data DR from becoming uneven as shown in Figure 3(c), and makes it possible to maintain the inter-class distances in the distribution ranges DrA to DrD uniform.

[0037] When training the feature extraction unit 13 and the class identification unit 15 using only the actual data DR as training data, the training unit 18 includes only the actual data DR in the same mini-batch and performs mini-batch training on the feature extraction unit 13 and the class identification unit 15. At this time, the training unit 18 calculates a loss function of the probability distribution output by the class identification unit 15, and trains the feature extraction model stored in the feature extraction model storage unit 12 and the template features stored in the template feature storage unit 14 so as to lower the loss function. When training only the feature extraction unit 13 using the virtual data DV as training data, the training unit 18 performs mini-batch training on the feature extraction unit 13 by including the real data DR and the virtual data DV in the same mini-batch. In this case, the training unit 18 trains the feature extraction unit 13 so that the data discrimination unit 16 cannot classify the features input from the feature extraction unit 13 as real data features or virtual data features. The real data DR and virtual data DV used when training the feature extraction unit 13 are examples of the "first real data" and "first virtual data" described in the claims.

[0038] For example, when the data input to the feature extraction unit 13 is virtual data DV, the learning unit 18 calculates the difference between the virtual data probability and the value "1" as the discrimination loss Lg, and when the data input to the feature extraction unit 13 is real data DR, the learning unit 18 calculates the difference between the virtual data probability and the value "0" as the discrimination loss Lg. To prevent the data discrimination unit 16 from classifying the data as real data features or virtual data features, the feature extraction unit 13 is trained so that the discrimination loss Lg becomes large. For example, the learning unit 18 calculates parameter update amounts for the feature extraction model to increase the discrimination loss Lg using a gradient method or a coordinate descent method with the discrimination loss Lg as an energy function, updates the feature extraction model by the calculated update amounts, calculates the discrimination loss Lg again, and evaluates the discrimination loss Lg. This process is repeated until an iteration termination condition is satisfied.

[0039] In this way, when the feature extraction unit 13 is trained so that the data identification unit 16 is unable to classify the features input from the feature extraction unit 13 as real data features or virtual data features, the training progresses so that the distribution of the features output from the feature extraction unit 13 reduces the deviation between the distribution range DvA to DvD (dashed line) of the features of the virtual data DV and the distribution range DrA to DrD (solid line) of the features of the real data DR. The arrows in Figure 3(d) show a schematic representation of this situation, indicating that the deviation between the distribution range DrA to DrD (solid line) of the features of the actual data DR and the distribution range DvA to DvD (dashed line) of the features of the virtual data DV becomes smaller.

[0040] In this way, when the deviation of the distribution range of the feature quantities of the virtual data DV from the distribution range of the feature quantities of the actual data DR is reduced, the recognition accuracy of the recognition device 1 when the actual data DR is input is improved. The reason for this will be explained with reference to Figures 4(a) and 4(b). In Figures 4(a) and 4(b), the distribution range Dr1 indicated by the solid line indicates the distribution range of the features of the actual data DR1 of a category that is frequently included in the training data, and the distribution range Dr2 indicated by the dashed-dot line indicates the distribution range of the features of the actual data DR2 of a category that is not frequently included in the training data.

[0041] For example, the category of real data DR1 may be face images of the learning target class taken from the front, and the category of real data DR2 may be profile images of the learning target class. If the amount of data contained in the learning data varies depending on the category, it may cause misrecognition of real data in categories with a small amount of data. The distribution range Dv indicated by the dashed line indicates the distribution range of the feature amount of the virtual data DV, and the dashed-dotted line Brv indicates the discrimination boundary of the data discrimination unit 16. The category of real data DR2 contains less data in the training data than the category of real data DR1, and so learning cannot be performed sufficiently. As a result, the intra-class variance of distribution range Dr2 is wider than the intra-class variance of distribution range Dr1. Furthermore, because virtual data DV is generated under several constraints, the intra-class variance of distribution range Dv is even wider than the intra-class variance of distribution range Dr2. As a result, the boundary line of distribution range Dr2 is located between the boundary line of distribution range Dr1 and the boundary line of distribution range Dv.

[0042] As the feature extraction unit 13 progresses in learning, the data discrimination unit 16 can no longer classify features as real data or virtual data. This reduces the deviation between the distribution range Dv of the features of the virtual data DV and the distribution range Dr1 of the features of the real data DR1. As a result, the feature extraction unit 13 learns so that the distribution range Dv of the features of the virtual data DV falls within the discrimination boundary Brv, as shown in Figure 4(b).

[0043] In this way, by learning to minimize the deviation between the distribution range Dv of the features of the virtual data DV and the distribution range Dr1 of the features of the actual data DR1, the feature extraction unit 13 also learns to minimize the deviation between the distribution range Dr2 of the features of the actual data DR2, which is between the distribution range Dv of the features of the virtual data DV and the distribution range Dr1 of the features of the actual data DR1, and the distribution range Dr1 of the actual data DR1. As a result, it is possible to extract features similar to those of the actual data DR1 of a category that is easy to collect from the actual data DR2 of a category that is difficult to collect as training data in practice. This prevents a decrease in recognition accuracy due to imbalances in the amount of training data caused by differences in categories, thereby improving recognition accuracy.

[0044] When the feature extraction unit 13 performs mini-batch training so that the data identification unit 16 cannot classify whether the data features are real data features or virtual data features, the virtual data DV to be included in the mini-batch together with the real data DR may be, for example, virtual data generated virtually based on the real data DR to be included in the mini-batch, or may be virtual data generated virtually based on real data of the same class as the class of the real data DR to be included in the mini-batch.

[0045] Furthermore, when the feature extraction unit 13 performs mini-batch training so that the data discrimination unit 16 cannot classify the data features as real data features or virtual data features, the virtual data DV to be included in the mini-batch together with the real data DR may be, for example, virtual data that is virtually generated based on real data of a class different from the class of the real data DR to be included in the mini-batch. The feature extraction unit 13 only needs to train so that the data discrimination unit 16 cannot classify the data features as real data features or virtual data features, and the real data DR and virtual data DV do not necessarily need to be trained using data of the same class.

[0046] In order to reduce the deviation of the distribution range of the features of the virtual data DV from the distribution range of the features of the real data DR output from the feature extraction unit 13 through such learning by the feature extraction unit 13, the data discrimination unit 16 needs to be an appropriate discriminator that can discriminate between real data features and virtual data features. For this reason, the learning unit 18 trains the data discrimination unit 16 so that it can distinguish whether the feature input from the feature extraction unit 13 is a real data feature or a virtual data feature (i.e., it learns the data discrimination model stored in the data discrimination model storage unit 17).

[0047] When training the data discrimination unit 16, the training unit 18 includes the real data DR and the virtual data DV in the same mini-batch, and performs mini-batch training on the data discrimination unit 16 so that it can distinguish whether the feature input from the feature extraction unit 13 is a real data feature or a virtual data feature. The real data DR and virtual data DV used when training the data discrimination unit 16 are examples of the "second real data" and "second virtual data" described in the claims.

[0048] For example, in order to train the data discrimination unit 16 so that it can classify data features as real data features or virtual data features, the data discrimination unit 16 may be trained so that the discrimination loss Lg is small. For example, the learning unit 18 calculates the parameter update amount of the data discrimination model to reduce the discrimination loss Lg by a gradient method or a coordinate descent method using the discrimination loss Lg as an energy function, updates the data discrimination model by the calculated update amount, calculates the discrimination loss Lg again, and evaluates the discrimination loss Lg. This process is repeated until an iteration termination condition is satisfied.

[0049] In addition, when the data identification unit 16 performs mini-batch training so that it can classify data features as real data or virtual data features, the virtual data DV to be included in the mini-batch together with the real data DR may be, for example, virtual data generated virtually based on the real data DR to be included in the mini-batch, or may be virtual data generated virtually based on real data of the same class as the class of the real data DR to be included in the mini-batch. However, virtual data generated virtually based on real data of a class different from the class of the real data DR included in the mini-batch cannot be used because if virtual data of a different class is included in the mini-batch and mini-batch learning is performed, the classification boundary Brv of the data classification unit 16 will be set between classes, and the above-mentioned effect will not be achieved.

[0050] To summarize the above explanation, the recognition device 1 of the embodiment learns through the following three learning processes. The feature extraction unit 13 and the class identification unit 15 are trained simultaneously using only the real data DR. The real data DR and the virtual data DV are included in the same mini-batch, and the feature extraction unit 13 performs mini-batch learning. The real data DR and the virtual data DV are included in the same mini-batch, and the data discrimination unit 16 performs mini-batch learning.

[0051] The mini-batch learning of the feature extraction unit 13 and the mini-batch learning of the data discrimination unit 16, which use the real data DR and the virtual data DV, may be performed simultaneously or alternately at different times. The learning frequencies of the feature extraction unit 13 and the data discrimination unit 16 may be equal or different. For example, one of the learning units may be performed M times (M is an integer equal to or greater than 2) and the other learning unit may be performed N times (N is an integer equal to or greater than 1 and smaller than M).

[0052] (operation) FIG. 5 is a flowchart illustrating an example of a learning method of the feature extractor 13 of the recognition device 1 according to an embodiment of the present invention. In step S1, the learning unit 18 trains the feature extraction unit 13 and the class identification unit 15 using only the real data DR and its loss function. In step S2, the feature extraction unit 13 extracts actual data feature amounts of the actual data DR.

[0053] In step S3, the feature extracting unit 13 extracts virtual data feature amounts of the virtual data DV. In step S4, the data identification unit 16 calculates the virtual data probability of the real data feature amount and also calculates the virtual data probability of the virtual data feature amount. In step S5, the learning unit 18 calculates the classification loss Lg of the virtual data probability calculated in step S4. In step S6, the learning unit 18 trains the feature extraction unit 13 so that the discrimination loss Lg calculated in step S5 becomes large, and then the process ends.

[0054] FIG. 6 is a flowchart of an example of a learning method for the data identification unit 16 of the recognition device 1 according to an embodiment of the present invention. In step S10, the feature extraction unit 13 extracts actual data feature amounts of the actual data DR. In step S11, the feature extracting unit 13 extracts virtual data feature amounts of the virtual data DV.

[0055] In step S12, the data identification unit 16 calculates the virtual data probability of the real data feature amount and also calculates the virtual data probability of the virtual data feature amount. In step S13, the classification loss Lg of the virtual data probability calculated in step S12 is calculated. In step S14, the learning unit 18 learns the data discrimination unit 16 so as to reduce the discrimination loss Lg calculated in step S13, and then the process ends.

[0056] (Effects of the embodiment)

[0057] (1) The recognition device 1 includes a feature extraction unit 13 that extracts features of input data and a class identification unit 15 that identifies the class of the input data based on the features extracted by the feature extraction unit. The learning unit 18 trains the feature extraction unit 13 and the class identification unit 15 so that the class identified by the class identification unit 15 based on the features extracted by the feature extraction unit 13 from actual data of a first class matches the first class. The feature extraction unit 13 extracts first virtual data features that are features of the first virtual data and first actual data features that are features of the first actual data from first virtual data, which is virtual data of the first class generated based on the actual data of the first class, and first actual data features that are features of the first actual data, respectively. The learning unit 18 trains the feature extraction unit 13 so that the data identification unit 16, which classifies features as actual data or virtual data, is unable to classify the first virtual data features into the first actual data features. This allows data from categories for which it is difficult to collect real data to be generated as virtual data and used as training data, thereby suppressing a decrease in recognition accuracy due to imbalances in the amount of training data caused by differences in categories and improving recognition accuracy. Furthermore, the recognition accuracy can be improved when training a recognition device using real data actually acquired from a class to be recognized and virtually generated virtual data as training data.

[0058] (2) The first virtual data and the first actual data may be included in the same mini-batch, and the feature extractor 13 may perform mini-batch learning. By performing mini-batch learning in this manner, the feature extractor 13 can be trained so that the data discriminator 16 is unable to classify the first virtual data feature and the first real data feature.

[0059] (3) The feature extraction unit 13 extracts second virtual data features, which are features of the second virtual data, and second actual data features, which are features of the second actual data, from second virtual data, which is virtual data of the second class generated based on actual data of the second class, and second actual data, which is actual data of the second class. The learning unit 18 trains the data classification unit 16 so that the second virtual data features and the second actual data features can be classified. By training the data discrimination unit 16 in this way, it is possible to configure a data discriminator that classifies the feature amounts of real data as those of virtual data. [Explanation of symbols]

[0060] 1...recognition device, 2...photographing unit, 3...communication unit, 4...storage unit, 5...image processing unit, 6...output unit, 7...operation input unit, 10...actual data acquisition unit, 11...virtual data generation unit, 12...feature extraction model storage unit, 13...feature extraction unit, 14...template feature storage unit, 15...class identification unit, 16...data identification unit, 17...data identification model storage unit, 18...learning unit, 19...identification result output unit

Claims

1. 1. A learning method for a recognition device including: a feature extraction unit that extracts features of input data; and a class identification unit that identifies a class of the input data based on the features extracted by the feature extraction unit, training the feature extraction unit and the class identification unit so that the class identified by the class identification unit based on the feature extracted by the feature extraction unit from actual data of a first class coincides with the first class; inputting first virtual data, which is virtual data of the first class generated based on actual data of the first class, and first actual data, which is actual data of the first class, to the feature extraction unit, and extracting first virtual data features, which are features of the first virtual data, and first actual data features, which are features of the first actual data, by the same feature extraction process; the feature extraction unit is trained so that a data classifier that classifies features of actual data or features of virtual data experiences a large discrimination loss when classifying features of actual data or features of virtual data, thereby reducing a distribution difference in a feature space between the first virtual data feature and the first actual data feature, and making it impossible to classify the first virtual data feature and the first actual data feature; A learning method characterized by:

2. The learning method according to claim 1 , wherein the first virtual data and the first real data are included in the same mini-batch to perform mini-batch learning for the feature extraction unit.

3. inputting second virtual data, which is virtual data of the second class generated based on actual data of the second class, and second actual data, which is actual data of the second class, to the feature extraction unit, and extracting second virtual data features, which are features of the second virtual data, and second actual data features, which are features of the second actual data, by the same feature extraction process; training the data classifier so as to improve classification accuracy between the second virtual data feature amount and the second real data feature amount; 3. The learning method according to claim 1 or 2.

4. 4. The learning method according to claim 3, wherein the second virtual data and the second real data are included in the same mini-batch to perform mini-batch learning of the data classifier.

5. 5. The learning method according to claim 3, wherein the feature extraction unit and the data classifier are trained simultaneously or separately.

6. A recognition device comprising the feature extraction unit and the class identification unit trained by the training method according to any one of claims 1 to 5.

Images