Learning method and object recognition device

Abstract

To perform learning of an object recognition apparatus using data obtained by data extension, as training data, while preventing fluctuation due to the data extension from being learned as a difference between classes.SOLUTION: A learning method of an object recognition apparatus comprising a feature quantity extraction unit which extracts feature quantities of input data and a class identifying unit which identifies a class of the input data on the basis of the feature quantities extracted by the feature quantity extraction unit, includes: acquiring first training data which is training data of a first class and second training data which is training data of a second class different from the first class (S1); generating first processed data and second processed data by applying the same data processing to the first training data and the second training data (S2); and performing mini-batch training of the feature quantity extraction unit by including the first processed data and the second processed data in the same mini-batch (S3).SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a learning method and an object recognition device. [Background technology]

[0002] Patent Document 1 describes a recognition system that includes a feature extraction means that extracts features of an input image, a class identification means that identifies the class of the input image based on the features extracted by the feature extraction means, and a learning means that trains the feature extraction means based on training images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-117565 Summary of the Invention [Problem to be solved by the invention]

[0004] Data augmentation, which adds random variations to training data, is known as a method for training a recognition system to extract features that are robust against variations in input data within the same class. For example, data augmentation for image data involves adding variations such as occlusion, lowering resolution, and lighting variations. When training a recognition device, the training proceeds so that the intra-class variance (i.e., the variance of features in the same class) in the feature space of features extracted by the feature extraction means is reduced and the inter-class variance (the variance of features between different classes) is increased. By training the recognition device using data that has been subjected to data augmentation as training data, the variation due to the data augmentation is learned as intra-class variation. As a result, the intra-class variance of features extracted by the feature extraction means is reduced.

[0005] However, if data augmentation is performed completely randomly and different variations are added to the data of different classes, there is a risk that the variations due to data augmentation will be learned as differences between classes. As a result, the variations due to data augmentation will be included in the differences between classes, which may widen the inter-class variance of features in an inappropriate direction. The present invention has been made in consideration of the above problems, and aims to prevent variations due to data augmentation from being learned as differences between classes when training an object recognition device using data that has been subjected to data augmentation as training data. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided a training method for an object recognition device including a feature extraction unit that extracts features of input data and a class identification unit that identifies a class of the input data based on the features extracted by the feature extraction unit. The training method includes acquiring first training data that is training data of a first class and second training data that is training data of a second class different from the first class, performing the same data processing on the first training data and the second training data to generate first processed data and second processed data, respectively, and including the first processed data and the second processed data in the same mini-batch to perform mini-batch training on the feature extraction unit.

[0007] According to another aspect of the present invention, there is provided an object recognition device comprising a feature extraction unit trained by the above-described training method. [Effects of the Invention]

[0008] According to the present invention, when training an object recognition device using data-augmented images as training images, it is possible to prevent variations due to data augmentation from being learned as differences between classes. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram illustrating an example of a hardware configuration of a recognition device according to an embodiment of the present invention. [Figure 2]1 is a block diagram illustrating an example of a functional configuration of a recognition device according to an embodiment of the present invention. [Figure 3] (a) and (b) are schematic diagrams showing the original images of the training images used to train person A, (c) and (d) are schematic diagrams showing the original images of the training images used to train person B, and (e) and (f) are schematic diagrams showing the original images of the training images used to train person C. [Figure 4] 3(a) to 3(f) are schematic diagrams showing processed images obtained by applying data extension to the images shown in FIG. 3(a) to 3(f), respectively. [Figure 5] FIG. 1 is a schematic diagram of an example of occlusion that hides the same part of different classes. [Figure 6] FIG. 10 is a schematic diagram of another example of occlusion that hides the same part of different classes. [Figure 7] 1 is a flowchart illustrating an example of a training method for a recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments of the present invention shown below are merely examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of the components to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0011] 1 is a schematic diagram showing an example of the hardware configuration of a recognition device 1 according to an embodiment of the present invention. The recognition device 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, an output unit 6, and an operation input unit 7. The first embodiment of the present invention is applicable to a recognition device that extracts feature amounts from an input image and performs class classification to identify the class of the input image based on the extracted feature amounts, or ID matching to identify a person in the input image. However, the present invention is not limited to this, and can also be applied to a recognition system that receives a speech signal as input data and recognizes the class of the input speech. For example, the present invention can be applied to a system that acquires a speech uttering a specific phrase as input speech and identifies the speaker.

[0012] The image capturing unit 2 is a surveillance camera installed for the purpose of monitoring a predetermined area, and is attached in a position where it can capture images of people staying within the area. The images captured by the image capturing unit 2 are transmitted to the image processing unit 5 via the communication unit 3. The communication unit 3 transmits and receives data between the imaging unit 2, the image processing unit 5, the output unit 6, and the operation input unit 7. A LAN (Local Area Network) or a public line such as the Internet can be used. The storage unit 4 is configured by a hard disk drive (HDD) or a solid state drive (SSD), and stores various programs including an operating system, and various data. The image processing unit 5 is composed of a CPU, GPU, MPU, peripheral circuits, terminals, various memories, etc., and transmits the results of image processing performed on the image captured by the imaging unit 2 to the output unit 6 via the communication unit 3.

[0013] The output unit 6 is a display, a projector, a printer, a removable drive, a USB (Universal Serial Bus) interface, a network interface, or the like that outputs various information generated by the recognition device 1. The operation input unit 7 is a mouse, keyboard, etc. that is operated by the user to accept inputs specifying templates and search ranges.

[0014] 2 is a block diagram of an example of the functional configuration of a recognition device 1 according to an embodiment of the present invention. The recognition device 1 is a device that acquires an image of a class to be recognized (hereinafter, may be referred to as a "class to be recognized") as input data, extracts features of the input data, and recognizes the class of the input data based on the extracted features. The recognition device 1 includes an input data acquisition unit 10, a data extension unit 11, a feature extraction model storage unit 12, a feature extraction unit 13, a template feature storage unit 14, a class identification unit 15, a learning unit 16, and an identification result output unit 17.

[0015] The photographing unit 2 or the memory unit 4 in FIG. 1 functions as an input data acquisition unit 10, the memory unit 4 functions as a feature extraction model memory unit 12 and a template feature memory unit 14, the image processing unit 5 functions as a data extension unit 11, a feature extraction unit 13, a class identification unit 15, and a learning unit 16, and the output unit 6 functions as an identification result output unit 17. First, the function of each unit of the recognition device 1 when executing processing to recognize the class of an input image will be described.

[0016] The input data acquisition unit 10 acquires an image of the recognition target class as input data. For example, the input data acquisition unit 10 may acquire an image obtained by capturing an image of the recognition target class using the capturing unit 2 as input data. The feature extraction model storage unit 12 stores a feature extraction model. For example, the feature extraction model may be modeled as a CNN configured with a multi-layer network such as that used in deep learning. In this case, the feature extraction model has a network structure in which multiple layers, such as convolutional layers, activation functions, and pooling layers, are connected in series, and the feature extraction model storage unit 12 stores information including the filter coefficients of the filters that make up the network and the network structure. These filter coefficients and other parameters of each layer are updated through learning.

[0017] The feature extraction unit 13 extracts features from the input data using the feature extraction model. That is, the feature extraction model is read from the feature extraction model storage unit 12, the image acquired by the input data acquisition unit 10 is input to this feature extraction model, and the feature output by the feature extraction model is output to the class identification unit 15.

[0018] The class identification unit 15 reads out template features from the template feature storage unit 14, and also receives features extracted from the input data by the feature extraction unit 13. The class identification unit 15 compares the data input from the feature extraction unit 13 with the template features read out from the template feature storage unit 14, and outputs a probability distribution indicating the probability that the class of the input data is each of the learning target classes. For example, the class identification unit 15 may be modeled using a CNN that is configured with a multi-layer network such as that used in deep learning.

[0019] The template feature storage unit 14 stores template features representing features of known classes. Hereinafter, the known classes whose template features are stored in the template feature storage unit 14 will be referred to as "learning target classes." For example, the communication unit 3 may receive template features of the learning target classes from outside and store the received template features in the template feature storage unit 14.

[0020] Furthermore, template features generated by template feature generating means (not shown) from input data that is known to be data of the learning target class may be stored in the template feature storage unit 14 . The class identification unit 15 outputs the calculated probability distribution to the learning unit 16 and the identification result output unit 17. The identification result output unit 17 outputs the identification result of the class identification unit 15, that is, the recognition result of the class of the input data.

[0021] Next, the function of each part of the recognition device 1 when the process of training the recognition device 1 is executed will be described. The input data acquisition unit 10 acquires images of the learning target class as learning images. For example, the input data acquisition unit 10 may acquire images obtained by photographing the learning target class with the photographing unit 2 as learning images. The input data acquisition unit 10 may acquire images of a known learning target class that have been prepared in advance as images to be used for learning and stored in the storage unit 4 as learning images. In the following description, the learning images of the learning target class acquired by the input data acquisition unit 10 may be referred to as "original images."

[0022] For example, Figures 3(a) and 3(b) are schematic diagrams of two different original images Ioa1 and Ioa2 of person A, which is a learning target class. Figures 3(c) and 3(d) are schematic diagrams of two different original images Iob1 and Iob2 of person B, which is a learning target class different from person A. Figures 3(e) and 3(f) are schematic diagrams of two different original images Ioc1 and Ioc2 of person C, which is a learning target class different from persons A and B. In the following description, original images Ioa1, Ioa2, Iob1, Iob2, Ioc1, and Ioc2 may be collectively referred to as "original image Io."

[0023] See Fig. 2. The data extension unit 11 performs data extension on the original image Io by performing data processing on the original image Io acquired by the input data acquisition unit 10. For example, the data extension unit 11 may perform occlusion to hide part of the original image Io as data processing. 4(a) to 4(f) are schematic diagrams of processed images Ipa1, Ipa2, Ipb1, Ipb2, Ipc1, and Ipc2 generated by applying occlusion to original images Ioa1, Ioa2, Iob1, Iob2, Ioc1, and Ioc2, respectively. In the following description, processed images Ipa1, Ipa2, Ipb1, Ipb2, Ipc1, and Ipc2 may be referred to as "processed image Ip."

[0024] The data augmentation unit 11 of this embodiment performs the same data processing on the original images of different classes to generate processed images of each class. Here, for example, the same data processing may be occlusion that hides the same parts of each class represented by the original image. For example, the processed images Ipa1, Ipb1, and Ipc1 shown in FIGS. 4(a), 4(c), and 4(e) may be generated by hiding the mouths of persons A, B, and C represented by original images Ioa1, Iob1, and Ioc1 shown in FIGS. 3(a), 3(c), and 3(e). The area around the mouth is prone to significant changes, such as an image captured at the moment the mouth is wide open, a hand placed over the mouth, or a mask being worn. This is done to enable robust feature extraction even when images of persons whose areas around the mouth have changed significantly are input.

[0025] Furthermore, for example, the processed images Ipa2, Ipb2, and Ipc2 shown in Figures 4(b), 4(d), and 4(f) may be generated by hiding the eyes of persons A, B, and C represented by original images Ioa2, Iob2, and Ioc2 shown in Figures 3(b), 3(d), and 3(f). For example, the area around the eyes is prone to change significantly in images captured at the moment the eyes are closed, or when wearing sunglasses or goggles. This is to enable learning to perform robust feature extraction even when images of persons with such significant changes around the eyes are input.

[0026] The data augmentation unit 11 may include a detector that detects identical parts of each class represented by the original image. The detector detects parts that are common to different classes (persons A, B, and C in the examples of FIGS. 3(a) to 3(f) and 4(a) to 4(f)) from the original image Io as identical parts. The data augmentation unit 11 applies occlusion to the original image Io by hiding the image areas of the identical parts detected by the detector. For example, the detector may model classes having a common structure based on rectangular information representing an object region or feature point information representing parts within the object, and use the model to detect regions representing the same part from the original image Io as image regions of the same part. Examples of models used for modeling include a face shape model having eyes, mouth, and nose as common structures, a box model having a rectangular parallelepiped structure, and a pose estimation model having joint points as common structures.

[0027] For example, when the original image is a face image of a person as in the examples of Figures 3(a) to 3(f), the detector may use a feature point extractor to model classes (persons A, B, and C in the examples of Figures 3(a) to 3(f)) into a face shape model having common structures such as "eyes," "mouth," and "nostrils," and detect common structural parts. For example, when the original image is a full image of an object (e.g., a "car" or a "suitcase") as in Figures 5(a) to 5(c), the detector may use a three-dimensional box object detector to model the classes (vehicles C1 to C3 in the example of Figures 5(a) to 5(c)) into a box model having a rectangular parallelepiped structure consisting of vertices v, sides s, and faces f, and detect common structural parts (e.g., the lower left front parts R1 to R3). For example, when the original image is a full image of a person as in Figures 6(a) to 6(c), the detector may use a pose estimator to model the class (persons P1 to P3 in the example of Figures 6(a) to 6(c)) into a pose estimation model having joint point j as a common structure, and detect common structural parts (e.g., legs).

[0028] The data extension unit 11 may add variations other than occlusion to the original image Io to generate the processed image Ip. For example, the data extension unit 11 may perform a "smoothing process" on the original images Io of different classes so that they become the same low-quality image to generate the processed image Ip, or may add a "brightness change" to generate the processed image Ip so that they have the same illumination variation. In the case of a recognition device 1 that recognizes classes of input speech, the same noise may be added to speech data of different classes, or the sound quality or speech rate may be changed to the same level. For example, in a system that identifies a speaker from speech that utters a specific phrase, noise may be added to parts that utter the same phrase, or the speech bit rate may be changed to produce the same low-quality speech. Alternatively, the speech rate may be changed to the same level by speech rate conversion, which converts the speech rate by an arbitrary factor while maintaining constant pitch and sound quality. The aforementioned "image smoothing processing" and "audio bit rate change" are examples of quality degradation as claimed in the claims. Also, "image brightness conversion" and "audio speech rate conversion" are examples of brightness change and rate change as claimed in the claims.

[0029] See Fig. 2. When training the recognition device 1 (i.e., when training the feature extraction model stored in the feature extraction model storage unit 12 and the template features stored in the template feature storage unit 14), the processed images generated by the data augmentation unit 11 are included in the same mini-batch as training images and input to the feature extraction unit 13. The feature extraction unit 13 extracts features of the processed images.

[0030] The class identification unit 15 outputs a probability distribution indicating the probability that the class of the processed image is each of the learning target classes, based on the feature amounts extracted from the processed image by the feature amount extraction unit 13. The learning unit 16 performs mini-batch learning of the feature extraction model stored in the feature extraction model storage unit 12 and the template features stored in the template feature storage unit 14 based on the probability distribution output by the class identification unit 15.

[0031] In training the feature extraction model, the training unit 16 calculates, for example, a loss function of the probability distribution output by the class identification unit 15, and trains the feature extraction model so as to lower the loss function. For example, it calculates parameter update amounts for the feature extraction model to reduce the loss function using a gradient method or coordinate descent method with the loss function as an energy function, updates the feature extraction model by the update amounts, and then calculates and evaluates the loss function again, repeating this process until an iteration termination condition is met.

[0032] In learning the template features, the learning unit 16 calculates, for example, a loss function of the probability distribution output by the class identification unit 15, and learns the template features stored in the template feature storage unit 14 so as to lower the loss function. For example, it calculates an update amount for the template features to reduce the loss function using a gradient method or coordinate descent method with the loss function as an energy function, updates the template features by the update amount, and then calculates and evaluates the loss function again, repeating this process until an iteration termination condition is met.

[0033] By training the recognition device 1 using processed images that have undergone data expansion in this way as training images, the variation due to data expansion is learned as variation within a class. As a result, the intra-class variance of the features extracted by the feature extraction unit 13 becomes smaller. Here, the processed images generated by the data augmentation unit 11 are images generated by performing the same data processing on original images of different classes. That is, processed images generated by performing the same data processing on original images of different classes are included in the same mini-batch, and the feature extraction model and template features are mini-batch trained. Therefore, the differences between the learned classes are less likely to include variations in data augmentation. As a result, the features are less affected by variations in data augmentation, and the features are dispersed within the feature space due to differences in features between classes. Therefore, compared to conventional methods where different data processing is performed for different classes, it is possible to expand the inter-class variance in the appropriate direction.

[0034] Note that the original image acquired by the input data acquisition unit 10 and the processed image generated by the data extension unit 11 (i.e., the processed image generated by performing the same data processing on original images of different classes) may be included in the same mini-batch as learning images to perform mini-batch learning of the feature extraction model and template features. In this case, the original image and the processed image are included in the same mini-batch and input to the feature extraction unit 13. In this way, by performing mini-batch learning by including the original image and the processed image generated from this original image in the same mini-batch, it becomes easier to learn the variation due to data expansion as variation within the class.

[0035] Note that the number of processed images included in the same mini-batch for the same class may be one or more. For example, the processed images of persons A, B, and C included in the same mini-batch may be only the original images Ipa1, Ipb1, and Ipc1, or only the processed images Ipa2, Ipb2, and Ipc2. All of the original images Ipa1, Ipa2, Ipb1, Ipb2, and Ipc2 may be included in the mini-batch.

[0036] Similarly, the number of original images included in the same mini-batch for the same class may be one or more. For example, the original images of persons A, B, and C included in the same mini-batch may be only original images Ioa1, Iob1, and Ioc1, or only original images Ioa2, Iob2, and Ioc2. All of original images Ioa1, Ioa2, Iob1, Iob2, and Ioc2 may be included in the mini-batch.

[0037] Furthermore, when mini-batch learning is performed by including multiple different processed images for the same class in the same mini-batch, processed images may be generated by applying different data augmentation to multiple different original images. For example, mini-batch learning may be performed by including in the same mini-batch the processed images Ipa1, Ipb1, and Ipc1 of Figures 4(a), 4(c), and 4(e) generated by hiding the mouth parts of the original images Ioa1, Iob1, and Ioc1 of Figures 3(a), 3(c), and 3(e), and the processed images Ipa2, Ipb2, and Ipc2 of Figures 4(b), 4(d), and 4(f) generated by hiding the eye parts of the original images Ioa2, Iob2, and Ioc2 of Figures 3(b), 3(d), and 3(f).

[0038] Furthermore, after the feature extraction model and template feature have been learned as described above, the feature output from the feature extraction unit 13 may be stored in the template feature storage unit 14 as a feature template. For example, the recognition device may be realized by installing the feature extraction model after learning and the feature templates extracted by the feature extraction unit 13 from images of the learning target class in separate hardware.

[0039] (operation) FIG. 7 is a flowchart of an example of a training method for the recognition device 1 according to the embodiment of the present invention. In step S1, the input data acquisition unit 10 acquires original images of a plurality of different learning target classes. In step S2, the data extension unit 11 generates processed images by performing the same data processing on the original images of the different learning target classes acquired in step S1.

[0040] In step S3, mini-batch learning is performed by including the edited images acquired in step S2 in the same mini-batch. Specifically, feature extraction unit 13 extracts features of the edited images acquired in step S2. Based on the features extracted by feature extraction unit 13, class identification unit 15 outputs a probability distribution indicating the probability that the class of the edited image is each of the learning target classes. Learning unit 16 calculates a loss function of the probability distribution and learns a feature extraction model and template features so that the loss function becomes low. The process then ends.

[0041] (Effects of the embodiment) (1) The recognition device 1 includes a feature extraction unit 13 that extracts features of input data, and a class identification unit 15 that identifies the class of the input data based on the features extracted by the feature extraction unit 13. The input data acquisition unit 10 acquires first training data, which is training data for a first class, and second training data, which is training data for a second class different from the first class. The data expansion unit 11 performs the same data processing on the first training data and the second training data to generate first processed data and second processed data, respectively. The feature extraction unit 13 and the class identification unit 15 are trained by mini-batch learning, in which the first processed data and the second processed data are included in the same mini-batch. This makes it difficult for variations in data augmentation to be included in the differences between classes being learned. As a result, since it is less susceptible to variations in data augmentation, the features are dispersed within the feature space due to differences in features between classes. Therefore, compared to conventional methods where different data processing is performed for different classes, it is possible to expand the inter-class variance in the appropriate direction.

[0042] (2) The feature extraction unit 13 and the class identification unit 15 may be trained by mini-batch training in which the first training data, the first processed data, the second training data, and the second processed data are included in the same mini-batch. This makes it easier to learn the variation due to data augmentation as variation within a class.

[0043] (3) The data processing may include at least one of occlusion, quality reduction, or value range change. This allows the above-mentioned effects (1) and (2) to be obtained in a recognition device that recognizes the class of an input image.

[0044] (4) As the same data processing, data processing may be performed on a portion of the first training data that corresponds to a predetermined location of the first class and a portion of the second training data that corresponds to the same predetermined location. For example, a detector may be used to detect portions corresponding to predetermined locations from the first learning data and the second learning data, and the same data processing may be performed on those portions. This allows the same data processing to be performed on the learning data.

[0045] (5) When the input data and the learning data are images representing a person, the same data processing may be performed on an image area corresponding to a specific part of the person in the first learning data and on an image area corresponding to the same part as the specific part in the second learning data. This allows the same data processing to be performed on the learning images in a recognition device that recognizes the class of an input image representing a person. [Explanation of symbols]

[0046] 1...recognition device, 2...photographing unit, 3...communication unit, 4...storage unit, 5...image processing unit, 6...output unit, 7...operation input unit, 10...input data acquisition unit, 11...data expansion unit, 12...feature extraction model storage unit, 13...feature extraction unit, 14...template feature storage unit, 15...class identification unit, 16...learning unit, 17...identification result output unit

Claims

1. A learning method for an object recognition device including: a feature extraction unit that extracts features of input data; and a class identification unit that identifies a class of the input data based on the features extracted by the feature extraction unit, acquiring first training data that is training data of a first class and second training data that is training data of a second class different from the first class; performing data processing on the first learning data and the second learning data such that the processing content and the setting value thereof are substantially the same to generate first processed data and second processed data, respectively; The first processed data and the second processed data are included in the same mini-batch, and the feature extraction unit and the class identification unit are subjected to mini-batch training. A learning method characterized by:

2. 2. The learning method according to claim 1, wherein the first training data, the first processed data, the second training data, and the second processed data are included in the same mini-batch, and the feature extraction unit and the class identification unit are subjected to mini-batch training.

3. The learning method according to claim 1 or 2, wherein the data processing includes at least one of occlusion, quality degradation, and value range change.

4. A learning method as described in any one of claims 1 to 3, characterized in that as the data processing in which the processing content and its setting value are substantially identical, the data processing in which the processing content is substantially identical is performed on a portion of the first learning data corresponding to a predetermined location of the first class and a portion of the second learning data corresponding to the same location as the predetermined location.

5. The learning method according to claim 4, characterized in that a detector is used to detect portions corresponding to the predetermined location from the first learning data and the second learning data, and the data processing is performed on the portions in a manner that is substantially the same as the processing content.

6. the input data and the learning data are images representing people, The learning method according to any one of claims 1 to 5, characterized in that the data processing in which the processing content and the setting values ​​are substantially the same is performed on an image area corresponding to a specified part of the person in the first learning data and an image area corresponding to the same part as the specified part in the second learning data.

7. An object recognition device comprising a feature extraction unit trained by the learning method according to any one of claims 1 to 6.

8. An object recognition device comprising a feature extraction unit and a class identification unit trained by the learning method according to any one of claims 1 to 6.

Images