Information processing device and control method thereof, and learning device

Abstract

To generate diverse data in feature space.SOLUTION: An information processing apparatus comprises: first obtaining means for obtaining one or more feature vectors associated with a first category from a vector group including a plurality of feature vectors each associated with a category; second obtaining means for obtaining one or more feature vectors associated with a second category, at least one of which is different from the first category, from the vector group; first calculation means for calculating a projection target vector from the feature vectors obtained by the first obtaining means; second calculation means for calculating a projection source vector from the feature vectors obtained by the second obtaining means; matrix calculation means for calculating a transformation matrix for transforming the projection source vector into the projection target vector; and generating means for generating a pseudo-feature vector associated with a third category different from the first category and the second category by transforming one or more feature vectors included in the feature vectors obtained by the second obtaining means using the transformation matrix.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a data generation technique. [Background technology]

[0002] In metric learning, which learns the distance between data, it is expected that the accuracy of learning can be improved by increasing the diversity of the learning data. Therefore, a method has been proposed in which data augmentation is performed in the feature space and applied to learning. Non-Patent Document 1 proposes a method of converting data belonging to a certain category into data belonging to another category in the feature space. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yuke Zhu, Yan Bai, Yichen Wei,"Spherical Feature Transform for Deep Metric Learning", In ECCV 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] In metric learning, the number of categories in the training data has a significant impact on performance. For example, in metric learning aimed at face recognition, the number of categories corresponds to the number of people included in the training data. However, the method in Non-Patent Document 1 can increase the number of data belonging to existing categories, but cannot increase the number of categories.

[0005] The present invention has been made in view of the above problems, and aims to provide a technique for generating diverse data in a feature space. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, an information processing device according to the present invention has the following arrangement: a first acquiring means for acquiring one or more feature vectors related to a first category from a vector group including a plurality of feature vectors each associated with a category; a second acquiring means for acquiring, from the group of vectors, one or more feature vectors, at least one of which is associated with a second category different from the first category; a first calculation means for calculating a projection vector from the feature vector acquired by the first acquisition means; a second calculation means for calculating a projection source vector from the feature vector acquired by the second acquisition means; a matrix calculation means for calculating a transformation matrix for transforming the source vector into the destination vector; a generating means for generating a pseudo feature vector associated with a third category different from the first category and the second category by performing a transformation process using the transformation matrix on one or more feature vectors included in the feature vector acquired by the second acquiring means; Equipped with. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a technique for generating diverse data in a feature space. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of a learning device according to the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating the functional configuration of an inference device according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating a learning set created in the learning set creation process. [Figure 5] 4 is a flowchart showing a learning process in the first embodiment. [Figure 6]10 is a detailed flowchart of a learning step process (S105). [Figure 7] 10 is a detailed flowchart of pseudo image feature vector generation (S205). [Figure 8] 4 is a flowchart showing an inference process in the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating the functional configuration of a learning device according to a second embodiment. [Figure 10] 10 is a flowchart showing a learning process in the second embodiment. [Figure 11] 10 is a detailed flowchart of a learning step process (S504). [Figure 12] 10 is a detailed flowchart of pseudo representative vector generation (S602). [Figure 13] FIG. 10 is a diagram illustrating the functional configuration of a learning device according to a third embodiment. [Figure 14] 10 is a flowchart showing a learning process in the third embodiment. [Figure 15] 10 is a detailed flowchart of pseudo representative vector generation (S801). [Figure 16] 10 is a detailed flowchart of a learning step process (S805). [Figure 17] FIG. 10 is a diagram illustrating the functional configuration of a learning device according to a fourth embodiment. [Figure 18] FIG. 10 is a diagram illustrating a learning set created in the learning set creation process. [Figure 19] 10 is a detailed flowchart of pseudo image feature vector generation. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) As a first embodiment of an information processing device according to the present invention, a learning device that generates a pseudo image feature vector based on training data including a plurality of image feature vectors and uses the pseudo image feature vector for training a neural network will be described below. An inference device that uses the neural network obtained by training will also be described. Below, first, the hardware configuration of the learning device and / or an information processing device that can be used in the inference device will be described with reference to FIG. 1. Then, the learning device will be described with reference to FIGS. 2 and 4 to 7, and the inference device will be described with reference to FIGS. 3 and 8.

[0011] <Hardware configuration> FIG. 1 is a diagram showing the hardware configuration of a computer device 100, which is an information processing device that can be used in a learning device and / or an inference device described below.

[0012] The CPU (Central Processing Unit) 101 controls the entire computer device 100. The ROM (Read Only Memory) 102 stores programs and parameters that do not require modification. The RAM (Random Access Memory) 103 temporarily stores programs and data supplied from external devices, etc. The external storage device 104 is a storage device such as a hard disk or memory card. The external storage device 104 may include optical disks such as flexible disks (FDs) and compact disks (CDs) that are detachable from the computer device 100, magnetic or optical cards, IC cards, memory cards, etc.

[0013] The input device interface 105 is an interface with the input device 109. The input device 109 is, for example, a pointing device or a keyboard that receives user operations and inputs data. The output device interface 106 is an interface with the monitor 110 that displays data held by the computer apparatus 100 and data supplied thereto.

[0014] The communication interface 107 is a communication interface for connecting to a network line 111 such as the Internet, and is connected to, for example, a network camera 112 via the network line 111. The network camera is an imaging device that generates video data by capturing images. The system bus 108 is a transmission path that communicatively connects each unit in the computer device 100. Each process described below is realized by the CPU 101 reading and executing a program stored in a computer-readable storage medium such as the ROM 102.

[0015] <Functional configuration of the learning device> 2 is a diagram showing the functional configuration of a learning device in the first embodiment. The learning device learns a machine learning model for face recognition. In this embodiment, an image feature vector representing the features of a face image of a person included in given learning data is transformed to generate an image feature vector representing a face image of a new person not included in the learning data, and the image feature vector is applied to learning.

[0016] The learning data storage unit 201 stores the learning data saved in the external storage device 104. The learning data includes a face image and a person ID. Here, the person ID is ID information that is the same for the same person.

[0017] The learning set creation unit 202 extracts multiple images of image data stored in the learning data storage unit 201 and creates a learning set. There are three types of learning sets: a positive set, a negative set, and a pseudo set. There are also three types of pseudo sets, which will be explained below. Figure 4 is a diagram illustrating the learning sets created in the learning set creation process.

[0018] The positive set 401 is made up of two facial images of the same person. In this embodiment, a plurality of positive sets are created for all possible combinations of facial images stored in the training data storage unit 201. The negative set 402 is made up of two facial images of different people. In this embodiment, a plurality of negative sets are created for all possible combinations of facial images stored in the training data storage unit 201. There are three pseudo sets: pseudo sets 403, 406, and 409.

[0019] The pseudo set 403 is composed of a first group 404 and a second group 405. The first group 404 includes one or more facial images associated with one person (same category). The second group 405 includes one or more facial images each of two or more people. The facial images included in the first group 404 and the second group 405 must be of different people (different categories). Furthermore, the two or more people included in the second group 405 are selected to be people with similar faces. One selection method is to assign an attribute label to each person in advance and select facial images of people with the same attribute. The attribute label may be race, gender, or a label combining race and gender. However, the selection method and the attribute labels used for selection are not limited to these.

[0020] The pseudo set 406 is composed of a first group 407 and a second group 408. The first group 407 includes one or more facial images of two or more people. The second group 408 includes one or more facial images of one person. The facial images included in the first group 407 and the second group 408 must be of different people. Furthermore, the two or more people included in the first group 407 are selected based on facial similarity. The selection method and attribute labels used for selection may be those described above for the pseudo set 403.

[0021] The Pseudo set 409 is composed of a first group 410 and a second group 411. The first group 410 and the second group 411 each include one or more facial images of two or more people, and at least one of the people is a different person. The two or more people included in each of the first group 410 and the second group 411 are selected from people with similar faces. The selection method and attribute labels used for selection may be those described above for the Pseudo set 403.

[0022] In this embodiment, a plurality of each of the three types of pseudo sets (pseudo set 403, pseudo set 406, pseudo set 409) are created as pseudo sets.

[0023] The learning set acquisition unit 203 acquires the learning set created by the learning set creation unit 202. The learning set acquisition unit 203 may also perform data processing known as data augmentation on the face images in the acquired learning set. For example, the data processing may change the color of the face image or add noise to pixel values. However, the data processing is not limited to these.

[0024] The vector generation unit 204 generates an image feature vector that represents the features of the image. A specific image feature vector generation method uses CNN (Convolutional Neural Networks), a type of neural network. CNN extracts abstract information from an input image by repeatedly performing a process consisting of convolution processing, activation processing, and pooling processing on the input image. In this case, the processing unit consisting of convolution processing, activation processing, and pooling processing is often called a layer.

[0025] There are several well-known activation processing methods used in this case, but a method called ReLU (Rectified Linear Unit) may be used, for example. There are also several well-known pooling processing methods, but a method called max pooling may be used, for example. For example, the CNN structure may be ResNet, as introduced in Reference 1, "K. He, X. Zhang, S. Ren, and J. Sun, "Identity mappings in deep residual networks," In ECCV 2016."

[0026] The first vector group acquisition unit 205 acquires image feature vectors generated from face images included in the first group of the Pseudo set acquired by the learning set acquisition unit 203. The second vector group acquisition unit 206 acquires image feature vectors generated from face images included in the second group of the Pseudo set acquired by the learning set acquisition unit 203.

[0027] The projection vector calculation unit 207 calculates a projection vector from the vector group acquired by the first vector group acquisition unit 205. Specific calculation methods include taking the average of the vector group, taking the average of only vectors sampled from the vector group, or using a weighted average instead of the average. However, the calculation method is not limited to these.

[0028] The projection source vector calculation unit 208 calculates the projection source vector from the vector group acquired by the second vector group acquisition unit 206. Specific calculation methods include taking the average of the vector group, taking the average of only vectors sampled from the vector group, or using a weighted average instead of the average. However, the calculation method is not limited to these.

[0029] The transformation matrix calculation unit 209 calculates the projection vector u obtained by the projection vector calculation unit 207. X , the projection source vector u obtained by the projection source vector calculation unit 208 Y Transformation matrix A that satisfies formula (1) is calculated for

[0030]

number

[0031] Specifically, first, as shown in Equation (2) and Equation (3), the Gram-Schmidt orthonormalization method is used to calculate u X , u Y is transformed into an orthonormal system.

[0032]

number

[0033]

number

[0034] Next, as shown in Equation (4), the transformation matrix A is calculated by applying the Rodrigues rotation formula. X and u Y It is the angle between.

[0035]

number

[0036] The pseudo vector generation unit 210 performs a transformation process on a vector arbitrarily selected from the vector group acquired by the second vector group acquisition unit 206, using the transformation matrix calculated by the transformation matrix calculation unit 209. In this way, a pseudo image feature vector (pseudo feature vector) is generated, and this is used as an image feature vector representing the face image of a new person.

[0037] The generation parameter update unit 211 updates the parameters for generating image feature vectors. Specifically, the parameters are updated so that the distance between image feature vectors representing the features of face images of the same person (same category) becomes smaller. The parameters are also updated so that the distance between image feature vectors representing the features of face images of other people (different categories) and the distance between any image feature vector and a pseudo-specific image feature vector become larger. As an updating method, for example, a loss function such as contrastive loss shown in Reference 2 "Raia Hadsell, Sumit Chopra, Yann LeCun, "Dimensionality Reduction by Learning an Invariant Mapping", In Proc. CVPR, 2006" is used.

[0038] First, the parameters are updated using the loss function L shown in Equation (5) so that the Euclidean distance D between the image feature vectors generated from the two face images included in the positive set becomes small.

[0039]

number

[0040] In addition, parameters are updated using the loss function L shown in Equation (6) so that the Euclidean distance D between the image feature vectors generated from the two face images included in the negative set increases, where m is a hyperparameter.

[0041]

number

[0042] Then, the parameters are updated using the loss function L shown in equation (6) so that the Euclidean distance D between the pseudo image feature vector and a vector arbitrarily selected from the vector group acquired by the first vector group acquisition unit 205 of the Pseudo set increases.

[0043] The update method and loss function to be used are not limited to these. Here, the generation parameters are weight coefficients of the CNN that constitutes the image feature generation unit 204, and are updated by backpropagation using the gradient of the loss function.

[0044] <Learning device operation> [Learning process] FIG. 5 is a flowchart showing the learning process in the first embodiment.

[0045] In S101 , the learning set creation unit 202 creates a positive set, a negative set, and a pseudo set as learning sets from image data included in the learning data held by the learning data holding unit 201 .

[0046] S102 is the start of the epoch loop. One loop used for all learning processes of the learning set created in S101 is called one epoch. The number of repeated epochs is determined in advance. To count the number of epoch repetitions, a variable i is used, which is first initialized to i=1. If the number of epochs is less than the predetermined number, the process moves to S103, and if the number of epochs exceeds the predetermined number, the process exits the loop and ends.

[0047] S103 is the start of the learning set loop. The learning sets created in S101 are assigned numbers in order starting from 1. To refer to these using the variable j, j is first initialized to 1. If j is equal to or less than the number of learning data, the process moves to S104, and if it exceeds the number of learning data, the process exits the loop and moves to S107.

[0048] In S104, the learning set acquisition unit 203 acquires one learning set from the learning sets created in S101. Alternatively, the image data included in the acquired learning set may be processed, such as by the aforementioned Data Augmentation, to create a learning set. Then, in S105, the learning device performs learning using the acquired data. Details of this process will be described later with reference to FIG. 6.

[0049] S106 is the end of the learning data loop, where 1 is added (incremented) to the variable j, and the process returns to S103. S107 is the end of the epoch loop, where 1 is added to the variable i, and the process returns to S102.

[0050] [Learning step processing] 6 is a detailed flowchart of the learning step process (S105). The following process is executed by the vector generation unit 204 to the generation parameter update unit 211 included in the learning device of FIG.

[0051] In S201, the learning device generates an image feature vector by performing forward processing of a neural network using a face image included in the acquired learning set as an input.

[0052] In S202, the learning device determines whether the acquired learning set is a positive set. If it is a positive set, the process proceeds to S203, and if not, the process proceeds to S204.

[0053] In S203, the learning device updates the generation parameters that control the generation of image feature vectors using Equation (5) so that the Euclidean distance between the two image feature vectors becomes smaller.

[0054] In S204, the learning device determines whether the acquired learning set is a pseudo set. If it is a pseudo set, the process proceeds to S205, and if not, the process proceeds to S206.

[0055] In S205, the learning device generates a pseudo image feature vector. This will be described in detail with reference to Fig. 7. Fig. 7 is a detailed flowchart of the pseudo image feature vector generation (S205).

[0056] In S301, the learning device acquires a first vector group and a second vector group. Specifically, the learning device acquires image feature vectors generated from face images included in the first group of the Pseudo set as the first vector group. The learning device also acquires image feature vectors generated from face images included in the second group of the Pseudo set as the second vector group.

[0057] In S302, the learning device calculates a projection target vector from the first vector group and a projection source vector from the second vector group. In S303, the learning device calculates the projection target vector u X , the projection vector u Y For the vectors, a transformation matrix A that satisfies the formula (1) is calculated. In S304, the learning device performs transformation using the transformation matrix A on any vector included in the second vector group, to generate a pseudo image feature vector.

[0058] In S206, the learning device updates the parameters that control the generation of image feature vectors using Equation (6) so that the Euclidean distance between the two image feature vectors increases.

[0059] Through the above learning process, the parameters that control the generation of image feature vectors are updated based on the pseudo image feature vectors generated from the image feature vectors.

[0060] <Functional configuration of the inference device> 3 is a diagram showing the functional configuration of an inference device in the first embodiment. Specifically, it shows the configuration of an inference device that performs inference of a main task using a neural network trained by the above-mentioned learning device. In this embodiment, the main task is face recognition, so the inference process involves determining whether two face images represent the same person.

[0061] The inference data acquisition unit 301 acquires two facial images. Here, we assume a system in which, when a user enters a login ID and password from a keyboard or the like for authentication, the system also uses a facial image acquired from a camera to verify the user's identity.

[0062] Specifically, a login screen is displayed on the monitor 110, and input of a login ID and password is accepted from the user via the input device 109. The login ID, password, and facial image are stored in advance in the external storage device 104, and when the login ID and password entered by the user match, a corresponding first facial image is acquired from the external storage device 104. Meanwhile, an instruction to have the user look at the NW camera 112 is displayed on the monitor 110, and the user is photographed via the NW camera 112 to acquire a second facial image. Note that the method of acquiring the two facial images and the method of configuring the system are not limited to these.

[0063] The vector generation unit 302 is the same as the vector generation unit 204 of the learning device 200. It holds the weights of the neural network trained by the learning device 200, and generates an image feature vector that represents the features of a face image.

[0064] The identity determination unit 303 calculates the similarity between the two facial images. In addition, it determines whether the person is the actual person based on the similarity. Specifically, it calculates the similarity between the image feature vectors of the two facial images obtained from the vector generation unit 302. If the similarity exceeds a pre-stored threshold, it determines that the person is the actual person.

[0065] The inference result output unit 304 outputs the inference result obtained by the identity determination unit 303. Specifically, it outputs the result of the login determination to the monitor 110. For example, if it is determined that the user is the identity of the user, it displays that the login was successful. If it is not determined that the user is the identity of the user, it displays that the login was unsuccessful. Furthermore, the result of the identity determination may be recorded in the external storage device 104. If a predetermined number of failures or more are recorded, it may be possible to notify the administrator user of the possibility of unauthorized access, for example. The method of outputting the inference result is not limited to these.

[0066] <Operation of the inference device> [Inference processing] 8 is a flowchart showing the inference processing in the first embodiment. The following processing is executed by the inference data acquisition unit 301 to the inference result output unit 304 included in the inference device of FIG.

[0067] In S401, the inference device acquires two facial images to be subjected to facial recognition via the inference data acquisition unit 301. In S402, the inference device obtains image feature vectors of the two facial images obtained in S401. In S403, the inference device performs identity determination using the two image feature vectors obtained in S402. Specifically, the Euclidean distance between the two image feature vectors is obtained as the similarity. If the obtained similarity exceeds a threshold stored in advance, it is determined that the person is the person in question. If it is equal to or less than the threshold, it is determined that the person is not the person in question (a different person).

[0068] In S404, the inference device outputs the determination result of S403 to the monitor 110 or the external storage device 104 via the inference result output unit 304. This allows the determination result to be communicated to the user or recorded.

[0069] <Effects> As described above, according to the first embodiment, facial images of people with similar faces (such as those with the same attributes) are grouped, and a pseudo image feature vector is generated from a group of image feature vectors generated from the grouped facial images. The pseudo image feature vector generated in this manner is a vector corresponding to other people who resemble the people included in the group. Therefore, by learning to be able to identify similar other people, the expressive ability of the image feature vector is improved, leading to improved accuracy in face authentication. Furthermore, even if there is a bias in the category distribution of data included in the training data stored in the training data storage unit 201, it is possible to prepare a dataset of image feature vectors that have a category distribution with a desired ratio (for example, a ratio that is in line with reality).

[0070] (Second embodiment) As a second embodiment of an information processing device according to the present invention, a learning device of another form will be described below as an example. Note that the hardware configuration and the functional configuration of the inference device are the same as those of the first embodiment, and therefore a description thereof will be omitted. Therefore, the learning device will be described below with reference to Figures 9 to 12.

[0071] <Functional configuration of the learning device> The learning device of this embodiment trains a machine learning model for face recognition using the "representative vector method" known from Literature 3, "Deng, et. al., "ArcFace: Additive Angular Margin Loss for Deep Face Recognition," In CVPR, 2019." The representative vector method is a face recognition learning method that sets feature vectors corresponding to each person included in the training data and uses these in combination to improve learning efficiency.

[0072] The representative vector is the fully connected layer W∈R adjacent to the output layer of the learning model when the number of people included in the learning data used is n. d×n is a vector that constitutes the representative vector W j ∈R d corresponds to the j-th person ID. The person ID is the ID information that will be the same if it is the same person. d is the number of dimensions of the representative vector. From the face image of the i-th person ID, the image feature vector generated using CNN is expressed as x i ∈R d The image feature vector x shown in Equation (7) i and the representative vector W j The distance θ between vectors calculated from the cosine similarity xiWj The parameters that make up the learning model are updated based on the above. More specifically, for the representative vector and image feature vector with the same person ID, θ xiWi On the other hand, for representative vectors and image feature vectors with different person IDs, Update the parameters so that TIFF0007765218000007.tif1422 is larger.

[0073]

number

[0074] By performing the above learning, the representative vector W j can be regarded as a vector representing the jth person ID in the feature space formed by the learning model.

[0075] In this embodiment, the representative vectors representing the people included in the training data are transformed. As a result, a group of pseudo representative vectors W′∈R representing new people not included in the training data is generated. d×n' and the fully connected layer W is {W,W'}∈R d×(n+n') and apply it to learning.

[0076] Figure 9 is a diagram showing the functional configuration of a learning device in the second embodiment. The basic configuration is the same as that of the first embodiment (Figure 2), but there are some differences. Specifically, a learning data acquisition unit 902 is provided instead of the learning set creation unit 202 and the learning set acquisition unit 203. A pseudo representative vector generation unit 910 is provided instead of the pseudo vector generation unit 210. The functions of the first vector group acquisition unit 905, second vector group acquisition unit 906, and generation parameter update unit 911 have been changed, and a representative vector holding unit 904 and representative vector update unit 912 are newly provided.

[0077] The learning data acquisition unit 902 acquires "face images" and "person IDs" to be used for learning from the learning data storage unit 901. The learning data acquisition unit 902 may perform data processing known as data augmentation, etc. For example, data processing may be performed by changing the color of the face image or adding noise to pixel values. Data processing is not limited to these. The representative vector storage unit 904 stores representative vectors that represent each person included in the learning data.

[0078] The first vector group acquisition unit 905 acquires several representative vectors from the representative vector holding unit 904 based on the person ID acquired by the learning data acquisition unit 902. Specifically, it acquires a "representative vector corresponding to the person of the person ID" and a "representative vector corresponding to a person whose face resembles the person of the person ID." A person whose face resembles the person of the person ID is determined, for example, by selecting a person whose representative vector is close to the representative vector corresponding to the person of the person ID in terms of inter-vector distance. Alternatively, the person may be determined by assigning an attribute label to each person in advance and selecting people with the same attribute. The attribute label may be race, gender, or a label combining race and gender. The selection method and attribute labels used for selection are not limited to these.

[0079] The second vector group acquisition unit 906 acquires several representative vectors from the representative vector holding unit 904 based on the person ID acquired by the learning data acquisition unit 902. Specifically, it acquires "representative vectors corresponding to an arbitrarily selected person different from the person of the person ID" and "representative vectors corresponding to a person whose face resembles the arbitrarily selected person." The selection of people whose faces resemble each other may be the same as in the case of the first vector group acquisition unit 905 described above.

[0080] The pseudo representative vector generation unit 910 generates a pseudo representative vector as a representative vector corresponding to the new person. Specifically, the pseudo representative vector is generated by performing a transformation process on a vector arbitrarily selected from the vector group acquired by the second vector group acquisition unit 906, using the transformation matrix calculated by the transformation matrix calculation unit 909.

[0081] The generation parameter update unit 911 updates the generation parameters. Specifically, the generation parameters are updated based on the image feature vectors generated from the face images acquired by the learning data acquisition unit 902, the representative vectors stored in the representative vector holding unit 904, and the pseudo representative vectors generated by the pseudo representative vector generation unit 910. At this time, the generation parameters are updated so that the image feature vectors generated from the face images approach the representative vectors corresponding to the person with the acquired person ID. The generation parameters are also updated so that the image feature vectors generated from the face images move away from the representative vectors corresponding to people other than the acquired person ID and the pseudo representative vectors generated by the pseudo representative vector generation unit 910.

[0082] As an updating method, for example, there is a method of updating parameters using a loss function L shown in Equation (8) by applying a loss function such as ArcFace. Details of ArcFace are given in the above-mentioned reference 3.

[0083]

number

[0084] N is the batch size, and s and m are hyperparameters. xiWi By adding m to the image feature vector x i is the representative vector W i If the vector is not located at a distance of m or more from the other representative vectors, it will not be considered the real person. In other words, the difficulty of learning can be improved.

[0085] The representative vector update unit 912 updates the representative vectors. Specifically, the representative vectors are updated so that the representative vectors corresponding to the person with the person ID acquired by the learning data acquisition unit 902 approach the image feature vectors generated from the face images acquired by the learning data acquisition unit 902. The representative vectors are also updated so that the representative vectors corresponding to people other than the person ID acquired by the learning data acquisition unit 902 move away from the image feature vectors generated from the face images acquired by the learning data acquisition unit 902.

[0086] As an updating method, for example, the parameters are updated using a loss function L shown in Equation (8) by applying a loss function such as ArcFace, similar to the above-described generation parameter updating unit 911. The updating method and loss function to be used are not limited to these.

[0087] <Learning device operation> [Learning process] FIG. 10 is a flowchart showing the learning process in the second embodiment.

[0088] S501 is the start of the epoch loop. The number of epochs to be repeated is predetermined. To count the number of epoch repetitions, a variable i is used, which is first initialized to i=1. If the number of epochs is less than the predetermined number, the process moves to S502. If the number of epochs exceeds the predetermined number, the process exits the loop and ends.

[0089] S502 is the start of the learning data loop. It is assumed that numbers are assigned sequentially starting from 1 to the learning data. This is referenced using the variable j, so j is first initialized to 1. If j is equal to or less than the number of learning data, the process moves to S503, and if it exceeds the number of learning data, the process exits the loop and moves to S506.

[0090] In S503, the learning data acquisition unit 902 acquires learning data. Furthermore, image data included in the acquired learning data may be processed, for example, by the aforementioned Data Augmentation, to generate the learning data. Then, in S504, the learning device performs learning using the acquired data. Details of this process will be described later with reference to FIG. 11.

[0091] S505 is the end of the learning data loop, where 1 is added (incremented) to the variable j, and the process returns to S502. S506 is the end of the epoch loop, where 1 is added to the variable i, and the process returns to S501.

[0092] [Learning step processing] 11 is a detailed flowchart of the learning step process (S504). The following process is executed by the vector generation unit 903 to the representative vector update unit 912 included in the learning device of FIG.

[0093] In S601, the learning device generates an image feature vector by forward processing of a neural network using a face image included in the acquired learning data as an input.

[0094] In S602, the learning device generates pseudo image feature vectors. This will be described in detail with reference to Fig. 12. Fig. 12 is a detailed flowchart of pseudo representative vector generation (S602).

[0095] In S701, the learning device acquires a first vector group based on the person ID (person A) acquired in S503. Specifically, the learning device acquires, as the first vector group, a "representative vector corresponding to the person ID (person A)" and a "representative vector corresponding to one or more person IDs similar to person A."

[0096] In S702, the learning device acquires a second vector group based on an arbitrary person ID (person B) not acquired in S503. Specifically, the learning device acquires, as the second vector group, a "representative vector corresponding to the person ID (person B)" and a "representative vector corresponding to one or more person IDs similar to person B."

[0097] In S703, the learning device calculates a projection target vector and a projection source vector. In S704, the learning device calculates a transformation matrix that satisfies Equation 1. In S705, the learning device generates a pseudo representative vector based on the transformation matrix.

[0098] In S603, the learning device updates the generation parameters that control the generation of image feature vectors using Equation 8. In S604, the learning device updates the representative vectors using Equation 8.

[0099] <Effects> As described above, according to the second embodiment, the pseudo image feature vectors described in the first embodiment are applied to learning by the representative vector method, which has been reported to have high recognition accuracy in machine learning for the purpose of face recognition. Using the updated representative vectors leads to further improvement in the accuracy of face recognition in the inference device.

[0100] (Third embodiment) As a third embodiment of an information processing device according to the present invention, a learning device of another form will be described below as an example. Note that the hardware configuration and the functional configuration of the inference device are the same as those of the first embodiment, and therefore a description thereof will be omitted. Therefore, the learning device will be described below with reference to Figures 13 to 16.

[0101] <Functional configuration of the learning device> The learning device of this embodiment learns a machine learning model for face recognition using the "representative vector method," as in the second embodiment. However, whereas in the second embodiment a pseudo representative vector is generated each time training data is acquired, in the third embodiment pseudo representative vectors are generated in advance and used as initial values ​​in training.

[0102] The pseudo representative vectors are generated as representative vectors that represent people who resemble people included in the training data. However, as the feature space changes as the training progresses, the people no longer resemble others. Therefore, in this embodiment, in order to maintain a similar state even when the feature space changes, the pseudo representative vectors are updated each time training data is acquired.

[0103] Fig. 13 is a diagram showing the functional configuration of a learning device in the third embodiment. The basic configuration is the same as that of the second embodiment (Fig. 9), but there are some differences. Specifically, it includes a representative vector holding unit 1304 and a pseudo representative vector updating unit 1314.

[0104] The pseudo representative vector holding unit 1311 holds the pseudo representative vectors generated in advance by the pseudo representative vector generating unit 1310 .

[0105] The pseudo representative vector update unit 1314 updates the pseudo representative vectors for the person included in the training data with the aim of "maintaining a state similar to the person." Specifically, it updates one or more pseudo representative vectors that are distributed in the vicinity of an image feature vector generated from a face image within a predetermined inter-vector distance so that they approach the image feature vector while maintaining a certain distance.

[0106] Image feature vector x i When updating so that K pseudo representative vectors distributed in the vicinity approach each other while maintaining a constant distance, the parameters are updated using a loss function such as ArcFace and the loss function L shown in Equation (9).

[0107]

number

[0108] N is the batch size, and s and m are hyperparameters. (W',xi,k) is the image feature vector x from the pseudo representative vector group W'. i For the kth nearest pseudo representative vector W j The above neighborhood is the vector distance θ calculated from the cosine similarity between the image feature vector and the pseudo representative vector shown in Equation (9). xiWj' The distance is determined by subtracting a constant value m from the inter-vector distance between the image feature vector and the pseudo representative vector, weakening the effect of the loss function and making it possible to maintain a constant interval. Note that the update method and loss function used are not limited to these.

[0109] <Device Operation> [Learning process] FIG. 14 is a flowchart showing the learning process in the third embodiment.

[0110] In S801, the learning device generates pseudo representative vectors, which will be described in detail later with reference to FIG.

[0111] S802 is the start of the epoch loop. The number of epochs to be repeated is predetermined. To count the number of epoch repetitions, a variable i is used, which is first initialized to i=1. If the number of epochs is less than the predetermined number, the process moves to S803. If the number of epochs exceeds the predetermined number, the process exits the loop and ends.

[0112] S803 is the start of the learning data loop. It is assumed that numbers are assigned sequentially starting from 1 to the learning data. This is referenced using the variable j, so j is first initialized to 1. If j is equal to or less than the number of learning data, the process moves to S804, and if it exceeds the number of learning data, the process exits the loop and moves to S807.

[0113] In S804, the learning data acquisition unit 1302 acquires learning data. Furthermore, image data included in the acquired learning data may be processed, such as by the aforementioned Data Augmentation, to generate the learning data. Then, in S805, the learning device performs learning using the acquired data. Details of this process will be described later with reference to FIG. 16.

[0114] S806 is the end of the learning data loop, where 1 is added (incremented) to the variable j, and the process returns to S803. S807 is the end of the epoch loop, where 1 is added to the variable i, and the process returns to S802.

[0115] [Pseudo representative vector generation process] 15 is a detailed flowchart of pseudo representative vector generation (S801). The following processing is executed by the learning data acquisition unit 1302 to pseudo representative vector generation unit 1310 included in the learning device of FIG.

[0116] S901 is the start of the loop for acquiring the first vector group. It is assumed that the representative vectors held in the vector holding unit 1304 are assigned numbers in order starting from 1. To refer to these vectors using a variable k, k is first initialized to 1. If k is less than or equal to the total number of representative vectors, the process moves to S902, and if it exceeds the total number, the process exits the loop and ends.

[0117] In S902, the learning device acquires a first vector group. Specifically, it acquires a "representative vector assigned to variable k" and "representative vectors corresponding to one or more persons who are thought to resemble the person to whom the representative vector assigned to variable k corresponds."

[0118] S903 is the start of a loop for acquiring the second vector group. The number of times acquisition is repeated is determined in advance. To count the number of times acquisition is repeated, a variable l is used, which is first initialized to l=1. If the number of times acquisition is less than the predetermined number, the process proceeds to S904. If the number of times acquisition is greater than the predetermined number, the process exits the loop and proceeds to S909.

[0119] In S904, the learning device acquires a second vector group. Specifically, it acquires "any one representative vector other than the representative vector assigned to variable k" and "representative vectors corresponding to one or more people who are thought to resemble the person to which the representative vector corresponds."

[0120] In S905, the learning device calculates a projection destination vector from the first vector group acquired in S902, and a projection source vector from the second vector group acquired in S904.

[0121] In S906, the learning device calculates the projection target vector u X , the projection vector u Y For the vectors, a transformation matrix A that satisfies the formula (1) is calculated. In S907, the learning device performs transformation on any vector included in the second vector group using the transformation matrix A to generate a pseudo representative vector.

[0122] S908 is the end of the loop for obtaining the second vector group, in which 1 is added to the variable l and the process returns to S903. S909 is the end of the loop for obtaining the first vector group, in which 1 is added to the variable k and the process returns to S901.

[0123] [Learning step processing] 16 is a detailed flowchart of the learning step process (S805). The following process is executed by the vector generation unit 1303 to the representative vector update unit 1315 included in the learning device of FIG.

[0124] In S1001, the learning device generates an image feature vector by forward processing of a neural network using a face image included in the acquired learning data as an input.

[0125] In S1002, the learning device updates the generation parameters that control the generation of image feature vectors using Equation (8). In S1003, the learning device updates the representative vectors using Equation (8). In S1004, the learning device updates the pseudo representative vectors using Equation (9).

[0126] <Effects> As described above, the third embodiment is applied to learning by the representative vector method, as in the second embodiment. However, unlike the second embodiment, pseudo representative vectors are generated in advance, and are not generated during learning. Therefore, it is possible to reduce the processing time required for learning compared to learning in the second embodiment.

[0127] (Fourth embodiment) As a fourth embodiment of the information processing device according to the present invention, a learning device of another form will be described below as an example. Note that the hardware configuration and the functional configuration of the inference device are the same as those of the first embodiment, and therefore a description thereof will be omitted. Therefore, the learning device will be described below with reference to Figures 17 to 19.

[0128] <Functional configuration of the learning device> The learning device of this embodiment trains a machine learning model for face recognition. In this embodiment, as in the first embodiment, a pseudo image feature vector representing a face image of a new person not included in the training data is generated and applied to learning. This aims to improve the authentication accuracy of face recognition.

[0129] In the first embodiment, similarity is determined based on the attribute labels assigned to people. However, even if people have the same attributes, they may not be treated as similar in the feature space, and in such cases, it is not possible to select an appropriate person. Therefore, in the fourth embodiment, whether or not the vector distance between image feature vectors is close is used as a criterion for determining similarity.

[0130] Fig. 17 is a diagram showing the functional configuration of a learning device in the fourth embodiment. The basic configuration is the same as that of the first embodiment (Fig. 2), but there are some differences. Specifically, the functions of the learning set creation unit 1702, first vector group acquisition unit 1706, and second vector group acquisition unit 1709 have been changed. In addition, an image feature vector holding unit 1705, a central vector acquisition unit 1707, and a neighborhood vector group acquisition unit 1708 have been added.

[0131] The learning set creation unit 1702 creates a learning set by extracting multiple images from the image data stored in the learning data storage unit 1701. There are three types of learning sets: a positive set, a negative set, and a pseudo set. Fig. 18 is a diagram illustrating the learning sets created in the learning set creation process.

[0132] The positive set 1801 is made up of two facial images of the same person. In this embodiment, a plurality of positive sets are created for all possible combinations of facial images stored in the learning data storage unit 1701. The negative set 1802 is made up of two facial images of different people. In this embodiment, a plurality of negative sets are created for all possible combinations of facial images stored in the learning data storage unit 1701. The pseudo set 1803 is made up of one facial image of any one person. A plurality of pseudo sets are created.

[0133] The image feature vector holding unit 1705 holds the image feature vectors generated by the vector generation unit 1704. The number of image feature vectors to be held is limited for each person ID, and the limit is determined in advance. If the limit is exceeded, the image feature vector that has been held the oldest from among the image feature vectors of that person ID is deleted.

[0134] The first vector group acquisition unit 1706 is composed of a central vector acquisition unit 1707 and a neighborhood vector group acquisition unit 1708. The central vector acquisition unit 1707 acquires the generated image feature vector from the vector generation unit 1704. Meanwhile, the neighborhood vector group acquisition unit 1708 selects, from the image feature vector holding unit 1705, one or more image feature vectors that are close to the image feature vector acquired by the central vector acquisition unit in terms of inter-vector distance. Note that the person ID of the face image represented by the selected image feature vector needs to be different from the person ID of the face image represented by the image feature vector acquired by the central vector acquisition unit.

[0135] The second vector group acquisition unit 1709 acquires a group of image feature vectors that are close to each other at an arbitrarily selected inter-vector distance from the image feature vector storage unit 1705. The selected image feature vectors represent a plurality of person IDs, and at least one person ID must be different from the person ID of the face image represented by the image feature vector acquired by the first vector group acquisition unit 1706.

[0136] <Device Operation> The learning process and learning step process in the fourth embodiment are the same as those in the first embodiment, and therefore will not be described again, except for the process of generating pseudo image feature vectors (S205).

[0137] [Pseudo image feature vector generation process] 19 is a detailed flowchart of pseudo image feature vector generation. The following processing is executed by the first vector group acquisition unit 1706 to pseudo vector generation unit 1713 included in the learning device of FIG.

[0138] In S1101, the learning device acquires the image feature vector generated in S201 as a central vector constituting the first vector group. In S1102, the learning device acquires a group of neighboring vectors constituting the first vector group from the image feature vector storage unit 1705. Specifically, the learning device acquires one or more image feature vectors that are near the image feature vector acquired in S1101 in terms of inter-vector distance as the group of neighboring vectors.

[0139] In S1103, the learning device acquires a second vector group. Specifically, the learning device acquires, from the image feature vector storage unit 1705, a group of image feature vectors that are close to each other at an arbitrarily selected inter-vector distance.

[0140] In S1104, the learning device calculates a projection target vector from the first vector group acquired in S1101 and S1102, and a projection source vector from the second vector group acquired in S1103. In S1105, the learning device calculates the projection target vector u X , the projection vector u Y In step S1106, the learning device performs a transformation using the transformation matrix A on any vector included in the second vector group to generate a pseudo representative vector. <Effects> As described above, according to the fourth embodiment, whether or not a person resembles another person is determined based on the inter-vector distance between image feature vectors representing face images in feature space. This configuration makes it possible to select more appropriate image feature vectors in the pseudo image feature vector generation process, and to generate more appropriate pseudo image feature vectors.

[0141] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0142] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0143] 201 Learning data storage unit; 202 Learning set creation unit; 203 Learning set acquisition unit; 204 Vector generation unit; 205 First vector group acquisition unit; 206 Second vector group acquisition unit; 207 Projection destination vector calculation unit; 208 Projection source vector calculation unit; 209 Transformation matrix calculation unit; 210 Pseudo vector generation unit; 211 Generation parameter update unit

Claims

1. a first acquiring means for acquiring one or more feature vectors related to a first category from a vector group including a plurality of feature vectors each associated with a category; a second acquiring means for acquiring, from the group of vectors, one or more feature vectors, at least one of which is associated with a second category different from the first category; a first calculation means for calculating a projection vector from the feature vector acquired by the first acquisition means; a second calculation means for calculating a projection vector from the feature vector acquired by the second acquisition means; a matrix calculation means for calculating a transformation matrix for transforming the source vector into the destination vector; a generating means for generating a pseudo feature vector associated with a third category different from the first category and the second category by performing a transformation process using the transformation matrix on one or more feature vectors included in the feature vector acquired by the second acquiring means; An information processing device comprising:

2. a vector generating means for generating a feature vector representing a feature of an image from the image; a storage means for storing, as the vector group, a plurality of feature vectors generated by the vector generation means based on a plurality of images; Further equipped 2. The information processing apparatus according to claim 1, wherein:

3. The vector generating means further includes an update means for updating a generation parameter used for generating a feature vector, The updating means updates the generation parameters so that the inter-vector distance between feature vectors belonging to the same category becomes smaller, the inter-vector distance between feature vectors belonging to different categories becomes larger, and the inter-vector distance between any feature vector included in the vector group and the pseudo feature vector becomes larger.

3. The information processing apparatus according to claim 2, wherein:

4. the vector generation means generates a feature vector from an image using a neural network; The generation parameters are the weighting coefficients of the neural network.

4. The information processing apparatus according to claim 3,

5. the first calculation means calculates the projection vector based on an average of the feature vectors acquired by the first acquisition means; the second calculation means calculates the projection source vector based on an average of the feature vectors acquired by the second acquisition means.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. a second storage means for storing a plurality of representative vectors, each of which is associated with a category and is a feature vector representative of the associated category, as a second vector group; the first acquisition means and the second acquisition means acquire a representative vector from the second vector group; The generating means generates a pseudo representative vector representing the third category.

6. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. The system further includes second update means for updating the second vector group held by the second holding means based on the pseudo representative vectors generated by the generating means.

7. The information processing apparatus according to claim 6,

8. a third updating means for updating the pseudo representative vectors generated by the generating means; The third updating means updates the pseudo representative vectors located within a predetermined inter-vector distance from any feature vector included in the vector group so that the inter-vector distance from the any feature vector becomes smaller.

8. The information processing device according to claim 6, wherein:

9. The categories correspond to people, and feature vectors associated with the same person belong to the same category.

2. The information processing apparatus according to claim 1, wherein:

10. The first category and the second category are assigned the same attribute label.

2. The information processing apparatus according to claim 1, wherein:

11. The attribute label is a label representing at least one of race or gender.

11. The information processing apparatus according to claim 10,

12. An information processing device according to any one of claims 1 to 11; a learning means for performing learning using at least the pseudo feature vector generated by the information processing device; A learning device comprising:

13. A control method for an information processing device, comprising: a first obtaining step of obtaining one or more feature vectors related to a first category from a vector group including a plurality of feature vectors each associated with a category; a second obtaining step of obtaining, from the group of vectors, one or more feature vectors, at least one of which is associated with a second category different from the first category; a first calculation step of calculating a projection vector from the feature vector acquired in the first acquisition step; a second calculation step of calculating a projection source vector from the feature vector acquired in the second acquisition step; a matrix calculation step of calculating a transformation matrix for transforming the source vector into the destination vector; a generating step of generating a pseudo feature vector associated with a third category different from the first category and the second category by performing a transformation process using the transformation matrix on one or more feature vectors included in the feature vector acquired in the second acquiring step; A control method comprising:

14. A program for causing a computer to execute the control method according to claim 13.

Images