Identification device, identification method, learning method, program, model and data structure

Abstract

To reduce the amount of operation for identification while suppressing reduction of an identification accuracy.SOLUTION: An identification device according to an aspect includes: extraction means for extracting an N-dimensional feature vector of a second format including an M-dimensional feature vector of a first format (M is an integer of at least 2 and N is an integer larger than M), from input data; and identification means for identifying an identification target of the input data on the basis of the first-format feature vector and the second-format feature vector.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a classification device, a classification method, a learning method, a program, a model, and a data structure. [Background technology]

[0002] Patent Document 1 discloses a face detection device that calculates a face surrounding likelihood using a second face detection algorithm in a detection area of ​​image data where the face likelihood calculated using a first face detection algorithm is equal to or greater than a threshold value α. The first face detection algorithm is a faster face detection algorithm than the second face detection algorithm, and the second face detection algorithm has a higher detection rate and a lower false detection rate for face image areas than the first face detection algorithm. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4903192 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to reduce the amount of calculation required for classification while suppressing a decrease in classification accuracy. [Means for solving the problem]

[0005] An identification device according to one embodiment of the present invention comprises an extraction means for extracting an N-dimensional (N is an integer greater than M) second-type feature vector including an M-dimensional (M is an integer greater than 2) first-type feature vector from input data, and an identification means for identifying an object to be identified in the input data based on the first-type feature vector and the second-type feature vector. [Effects of the Invention]

[0006] According to one aspect of the present invention, it is possible to reduce the amount of calculation required for classification while suppressing a decrease in classification accuracy. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an identification device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing a detailed configuration example of the identification device according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing a schematic flow of processing by the identification unit in FIG. 2; [Figure 4] FIG. 2 is a diagram showing an example of the configuration of a feature vector according to the first embodiment. [Figure 5] 4 is a flowchart showing a classification process according to the first embodiment. [Figure 6] 10 is a flowchart showing a classification process according to the second embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the configuration of a learning device according to a third embodiment. [Figure 8] FIG. 10 is a block diagram showing an example of the configuration of an identification device according to a fourth embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a model used for feature extraction according to the fifth embodiment. [Figure 10] FIG. 13 is a block diagram showing an example of the hardware configuration of an identification device according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). The technical scope of the present invention is determined by the claims and is not limited by the individual embodiments below.

[0009] First Embodiment In the following description, an object identification device using face recognition will be taken as an example of an identification device. However, this embodiment is not limited to object identification devices using face recognition, and can also be applied to pet recognition such as dogs and cats, and object recognition such as automobiles and bicycles. Furthermore, this embodiment may be applied to image recognition devices, character recognition devices, or voice recognition devices other than object identification devices, and may be applied to any identification device that performs identification using feature vectors.

[0010] Fig. 1 is a block diagram showing an example of the configuration of an identification device according to a first embodiment. In the example of Fig. 1, a face recognition device is shown as the identification device. The face recognition device can be used to identify a person captured on camera in a surveillance camera system among pre-registered people, or to perform face recognition at an entrance / exit gate in an entrance / exit management system.

[0011] For the software-implemented functions of the functional blocks shown in FIG. 1, a program for providing the function of each functional block is stored in a memory such as a read-only memory (ROM). The program is then loaded into a random-access memory (RAM) and executed by a central processing unit (CPU). For the hardware-implemented functions, a dedicated circuit may be automatically generated on an FPGA from a program for implementing the function of each functional block using, for example, a predetermined compiler. FPGA stands for Field Programmable Gate Array. Alternatively, a gate array circuit may be formed in a similar manner to an FPGA to implement the function as hardware. Alternatively, the function may be implemented using an application-specific integrated circuit (ASIC). Note that the functional block configuration shown in FIG. 1 is merely an example; multiple functional blocks may constitute a single functional block, or one functional block may be divided into blocks that perform multiple functions.

[0012] In FIG. 1, the face recognition device includes a detection unit 1, a feature extraction unit 2, and a classification unit 3. The detection unit 1 detects a recognition target, i.e., a face region, from an input image 11. The detection unit 1 can detect a face region using, for example, MTCNN. MTCNN stands for Multi-task Cascaded Convolutional Neural Networks for Face Detection, based on TensorFlow.

[0013] The feature extraction unit 2 extracts feature amounts from the face image detected by the detection unit 1. The feature amounts can be expressed as feature vectors 12A. In face recognition processing, the feature vectors 12A are designed to be able to accurately determine whether or not the faces represent the same person. In the case of face recognition, the feature vectors 12A take similar values ​​if the faces represent the same person, regardless of facial expression, shooting direction, lighting, or the presence or absence of accessories, and take significantly different values ​​if the faces represent different people.

[0014] The identification unit 3 compares the feature vector 12A extracted by the feature extraction unit 2 with a pre-registered feature vector 12B. Then, an identification process S13 identifies which of the pre-registered feature vectors 12B the feature vector 12A extracted by the feature extraction unit 2 is closest to. At this time, the identification unit 3 pre-registers feature vectors 12B extracted from images of many known people, and can compare the feature vector 12A extracted by the feature extraction unit 2 with each of the many pre-registered feature vectors 12B.

[0015] Here, feature vectors 12A and 12B are N-dimensional (N is an integer greater than M) second-format feature vectors including M-dimensional (M is an integer greater than or equal to 2) first-format feature vectors. Then, the identification unit 3 identifies a face based on the first-format feature vectors and the second-format feature vectors. The identification unit 3 calculates a first similarity between the first-format feature vectors, and can calculate a second similarity if the first similarity is greater than a predetermined value. The first similarity can have a higher recall rate than the second similarity, and the second similarity can have a higher matching rate than the first similarity. The recall rate is the identification rate for input images of the same object, and in face recognition, it is the proportion of cases where two images are determined to be the same person. The matching rate is the proportion of cases where two images are correctly identified as the same object, and in face recognition, it is the proportion of cases where two images are determined to be the same person that are actually the same person.

[0016] Generally, when calculating the similarity between feature vectors of many people, the feature vectors are almost always of different people. In this case, the feature vectors of different people are more different than the feature vectors of similar people. Therefore, when determining whether or not the people are different, the accuracy of the determination can be ensured even if the number of dimensions of the feature vectors is reduced compared to when determining whether or not the people are the same. Therefore, the identification unit 3 eliminates feature vectors of different people from the feature vectors of many people based on the first similarity and identifies feature vectors of similar people. Then, the identification unit 3 identifies feature vectors of people who are truly identical from the feature vectors of similar people based on the second similarity. This eliminates the need for the identification unit 3 to calculate the similarity between high-dimensional feature vectors of many people, including different people, and can reduce the amount of calculation required for identification while suppressing a decrease in identification accuracy.

[0017] FIG. 2 is a block diagram showing a detailed configuration example of the identification device according to the first embodiment. 2, classification device 30 includes image input unit 31, detection unit 32, feature extraction unit 33, classification unit 34, and feature storage unit 35. Classification unit 34 includes matching unit 340. Matching unit 340 includes first similarity calculation unit 341 and second similarity calculation unit 342. Note that detection unit 32, feature extraction unit 33, and classification unit 34 can be used as detection unit 1, feature extraction unit 2, and classification unit 3, respectively, in FIG. 1.

[0018] 1 to the detection unit 32. The image input unit 31 is, for example, an image capturing device such as a camera or an image recording device. The detection unit 32 detects an object to be identified (here, a face image) from the input image 11 and outputs it to the feature extraction unit 33. The feature extraction unit 33 extracts a feature vector 12A from the face image detected by the detection unit 32. The feature storage unit 35 stores feature vectors 12B extracted in advance from images of a large number of known people.

[0019] The identification unit 34 identifies which of the feature vectors 12B stored in the feature storage unit 35 the feature vector 12A extracted by the feature extraction unit 33 is closest to. The matching unit 340 matches the feature vectors 12A and 12B with each other and calculates the similarity between the feature vectors 12A and 12B. The first similarity calculation unit 341 calculates a first similarity between the feature vectors of the first format between the feature vectors 12A and 12B. The second similarity calculation unit 342 calculates a second similarity between the feature vectors of the second format between the feature vectors 12A and 12B. At this time, the second similarity calculation unit 342 can calculate the second similarity if the first similarity is greater than a predetermined value.

[0020] Fig. 3 is a diagram showing a schematic flow of processing by the identification unit in Fig. 2. Note that Fig. 3 shows an example in which feature vectors 12A and 12B are matched with each other without distinguishing between the first format and the second format of each feature vector 12A and 12B.

[0021] In step S21 of Fig. 3, the classification unit 34 of Fig. 2 receives two feature vectors, that is, a feature vector 12A extracted from the input image 11 and one of the pre-registered feature vectors 12B, and calculates the similarity between the two feature vectors. The pre-registered feature vector 12B can include the corresponding person information 23.

[0022] The similarity takes a large value if the feature vectors 12A and 12B are close to each other and belong to the same person, and a small value if the feature vectors 12A and 12B belong to different people. Note that distance may be used instead of similarity, but they have the same meaning. The distance takes a value close to 0 if the feature vectors 12A and 12B are close to each other, and takes a large value (absolute value) if the feature vectors 12A and 12B are far from each other. In this specification, the explanation will be given using similarity, but distance may also be used.

[0023] Next, in step S22, the identification unit 34 calculates the similarity between all registered feature vectors 12B and the feature vector 12A, and then calculates the maximum similarity 22, which is the maximum value of the similarity.

[0024] Next, in step S23, the classification unit 34 compares the maximum similarity 22 with a predetermined value. If the maximum similarity 22 is greater than the predetermined value, the classification unit 34 proceeds to step S24. In step S24, the classification unit 34 determines that a corresponding person exists, and identifies (identifies) the person corresponding to the maximum similarity 22 as the person appearing in the input image 11. On the other hand, if the maximum similarity 22 is equal to or less than the predetermined value, the classification unit 34 proceeds to step S25. In step S25, the classification unit 34 determines that the feature vector 12B is different from any of the registered people, and determines that a corresponding person does not exist.

[0025] 3, the classification unit 34 calculates the maximum similarity after calculating all similarities of the registered feature vectors 12B, but the maximum similarity value may be updated each time a similarity is calculated. Also, in order to reduce the number of times of matching in step S21, clustering of the registered feature vectors 12B may be performed.

[0026] Here, the matching process in step S21 is performed the same number of times as the number of registered feature vectors 12B. Therefore, the number of matching processes in step S21 is positively correlated with the number of registered feature vectors 12B. As the number of registered feature vectors 12B increases, the number of matching processes in step S21 increases, and the average processing time for matching in S21 becomes longer. Furthermore, as the number of registered feature vectors 12B increases, the storage capacity of the feature storage unit 35 in FIG. 2 also increases.

[0027] Here, in order to reduce the amount of calculation for matching in step S21 and to prevent an increase in the storage capacity of the feature storage unit 35, the feature vectors 12A and 12B are represented as N-dimensional second-type feature vectors including M-dimensional first-type feature vectors. Then, the identification unit 34 identifies a face based on the first-type feature vectors and the second-type feature vectors.

[0028] FIG. 4 is a diagram showing an example of the configuration of a feature vector according to the first embodiment. In FIG. 4, each of the feature vectors 12A and 12B has a data structure in which an M-dimensional feature vector 41 in a first format is included in an N-dimensional feature vector 42 in a second format.

[0029] Each of the feature vectors 12A and 12B is an N-dimensional second-type feature vector 42, but when focusing on a part of it, that is, M dimensions, it can also be treated as an M-dimensional first-type feature vector 41. In other words, when two feature vectors 12A and 12B are provided to the matching unit 340, the matching unit 340 can calculate a first similarity between the first-type feature vectors 41, and can also calculate a second similarity between the second-type feature vectors 42.

[0030] At this time, the first-form feature vector 41 is set so that the recall of the classification target (here, a face) by the first-form feature vector 41 is higher than the recall of the classification target by the second-form feature vector 42. Also, the second-form feature vector 42 is set so that the compatibility of the second-form feature vector 42 with respect to the classification target is higher than the compatibility of the first-form feature vector 41 with respect to the classification target.

[0031] As a result, the identification unit 34 can exclude second-type feature vectors 42 that include first-type feature vectors 41 of different people while retaining second-type feature vectors 42 that include first-type feature vectors 41 of the same person based on matching between first-type feature vectors 41. The identification unit 34 can then identify the same person based on matching between second-type feature vectors 42 from which second-type feature vectors 42 that include first-type feature vectors 41 of different people have been excluded. In this case, the identification unit 34 does not need to perform matching with second-type feature vectors 42 that include first-type feature vectors 41 of different people in order to identify the same person, and the amount of calculation required for matching can be reduced.

[0032] FIG. 5 is a flowchart showing the classification process according to the first embodiment. In step S51 of FIG. 5, the first similarity calculation unit 341 of FIG. 2 compares the feature vectors 41 of the first format out of the feature vectors 12A and 12B, and calculates a first similarity 51.

[0033] Next, in step S52, the identification unit 34 determines whether the first similarity 51 is greater than a predetermined value. If the first similarity 51 is greater than the predetermined value, the identification unit 34 proceeds to step S53. In step S53, the second similarity calculation unit 342 compares the second-format feature vectors 42 of the feature vectors 12A and 12B with each other, calculates a second similarity 53, and sets this as the comparison result of step S21 in FIG. 3. Here, the second similarity calculation unit 342 compares the second-format feature vectors 42 of the feature vector 12A, among the feature vectors 12B held in the feature holding unit 35, that include a first-format feature vector 41 whose first similarity 51 is greater than the predetermined value.

[0034] At this time, if it is determined that the two people are not the same person based on the first similarity 51, the identification unit 34 does not calculate the second similarity 52. ​​Generally, when calculating similarities with feature vectors of many people, it is determined that the two people are different people in most cases. For this reason, the identification unit 34 can omit calculating the second similarity 52 for almost all of the feature vectors 12B stored in the feature storage unit 35. For this reason, the identification unit 34 can shorten the average time required to calculate the similarity between the feature vectors 12A and 12B compared to when calculating the similarity between all of the feature vectors 12B stored in the feature storage unit 35.

[0035] Step S52 is a process for reducing the number of calculations of second similarity 52 in step S53. At this time, in order to correctly calculate the similarity of the same person while reducing the number of calculations of second similarity 52, it is necessary to reliably calculate second similarity 53 when the people are the same. For this reason, the feature vectors 12B for which second similarity 53 is calculated may include feature vectors 12B of a small number of non-identical people. For this reason, the first-type feature vectors 41 are designed to increase the recall rate to 100%, in principle, and the second-type feature vectors 42 are designed to increase the degree of matching.

[0036] If the first similarity 51 is equal to or less than the predetermined value, the classification unit 34 proceeds to step S54. In step S54, the classification unit 34 corrects the first similarity 51 and sets the corrected result as the matching result of step S21 in FIG. The similarity correction is a correction that makes the value of the first similarity 51 comparable in the same way as the second similarity 53. Examples of this correction include a method of setting the second similarity 53 to a small, constant value that is unlikely to be the same person, and a method of multiplying the first similarity 51 by a constant so that the small, constant value that is unlikely to be the same person is set as the upper limit of the second similarity 53. Either method may be used for similarity correction, but as shown in FIG. 3, if only the maximum similarity 22 is used as the matching result in step S21, it is sufficient to set the second similarity 53 to a small, constant value that is unlikely to be the same person.

[0037] As described above, according to the first embodiment, the feature vectors 12A and 12B are represented by an N-dimensional second-format feature vector including an M-dimensional first-format feature vector. The classification unit 34 then classifies the classification target based on the first-format feature vector and the second-format feature vector, thereby reducing the amount of calculation required for matching feature vectors and suppressing an increase in the storage capacity required for registering feature vectors.

[0038] Second Embodiment 6 is a flowchart showing the classification process according to the second embodiment. In the first embodiment, when the second similarity 53 is calculated in step S53, the first similarity 51 is calculated in step S51. If the calculation result of the first similarity 51 is not used in calculating the second similarity 53 in step S53, the processing time increases by the time required to calculate the first similarity 51. Therefore, in the second embodiment, to reduce the processing time, an intermediate result of calculating the first similarity 51 is used in calculating the second similarity 53.

[0039] The process in Fig. 6 differs from the process in Fig. 5 in that an intermediate result 61 of the calculation of the first similarity 51 in step S51 is used in the calculation of the second similarity 52 in step S53. In this case, the similarity between the feature vectors can be, for example, cosine similarity. Here, if two feature vectors for which similarity is to be calculated are Fx and Fy, each of the feature vectors Fx and Fy can be expressed as an N-dimensional vector including an M-dimensional vector, as shown in the following equations (1) and (2).

[0040]

number

[0041]

number

[0042] When the first similarity 51 is expressed as a cosine similarity SA1 of an M-dimensional vector, it is given by the following equation (3), and when the second similarity 52 is expressed as a cosine similarity SA2 of an N-dimensional vector, it is given by the following equation (4).

[0043]

number

[0044]

number

[0045] Here, α, β, γ, δ, ε, and ζ are defined as shown in the following equations (5) and (6).

[0046]

number

[0047]

number

[0048] In this case, the cosine similarity SA1 of the M-dimensional vector is given by the following equation (7) using α, β, and γ.

[0049]

number

[0050] The cosine similarity SA2 of an N-dimensional vector is given by the following equation (8) using α, β, γ, δ, ε, and ζ.

[0051]

number

[0052] α, β, and γ are used both in calculating the first similarity 51 in step S51 and in calculating the second similarity 52 in step S53. Therefore, the second similarity calculation unit 342 uses the values ​​α, β, and γ calculated by the first similarity calculation unit 341 as intermediate result 61 in calculating the second similarity 52. ​​As a result, the matching in step S21 in Fig. 3 requires only one additional calculation for multiplication, division, and square root, compared to when calculating both the first similarity 51 and the second similarity 52.

[0053] Furthermore, the identification device 30 can pre-calculate 1 / √β, 1 / √γ, 1 / √(β+ε), and 1 / √(γ+ζ) during feature extraction and add them to the feature vector. In this case, the cosine similarity SA1 of an M-dimensional vector is given by the following equation (9), and the cosine similarity SA2 of an N-dimensional vector is given by the following equation (10). As a result, the amount of calculation required for matching in step S21 is reduced to just two additional multiplications.

[0054]

number

[0055]

number

[0056] In the first embodiment, in most cases of matching in step S21 of Fig. 3, the second similarity 52 is not calculated. However, if a large number of similar feature vectors 12B are registered, the processing time required to calculate the first similarity 51 increases. In the second embodiment, even if a large number of similar feature vectors 12B are registered, the increase in the time required to calculate the similarity can be suppressed.

[0057] In the above description, an example has been given in which cosine similarity is used as the similarity, but other indices, such as Manhattan distance or Euclidean distance, may also be used.

[0058] <Third embodiment> Each of feature vectors 12A and 12B has a high recall rate in the M-dimensional first format and a high discrimination performance in the N-dimensional second format. By designing each of feature vectors 12A and 12B in this manner, the amount of calculation required to discriminate between the same person can be reduced compared to when feature vectors are designed to only improve the discrimination performance in the N-dimensional second format. In this case, any feature extraction unit may be used as feature extraction unit 33 in Figure 2 as long as feature vector 12A having a high recall rate in the M-dimensional first format and a high discrimination performance in the N-dimensional second format can be extracted from a facial image.

[0059] For example, a deep neural network may be used to generate the feature extraction unit 33. There are various methods for training a deep neural network for object identification, but in the third embodiment, the Siamese Network method [Bromley, 1993] is used as an example.

[0060] FIG. 7 is a block diagram showing an example of the configuration of a learning device according to the third embodiment. 7, the learning device 7 includes a DNN (Deep Neural Network) unit 72 and a weight update unit 74. The DNN unit 72 includes DNNs 721 and 722. The DNNs 721 and 722 have the same configuration. The weight update unit 74 includes a first loss function 741 and a second loss function 742.

[0061] An image pair 71, which is a pair of images 711 and 712, is input to the DNN unit 72, and a feature vector pair 73, which is a pair of N-dimensional feature vectors 731 and 732, is output from the DNN unit 72. At this time, the image pair 71 corresponds to the feature vector pair 73. A label T (not shown) indicating whether the images are of the same person or not is separately assigned to the image pair 71, and takes the value 1 if they are the same person, and 0 if they are not the same person. The image pair 71 to which this label T is separately assigned can be used as training data for training each of the DNNs 721 and 722.

[0062] The learning device 7 acquires training data to train each of the DNNs 721 and 722. An image 711 is input to the DNN 721, and an image 712 is input to the DNN 722. Then, each of the DNNs 721 and 722 extracts an N-dimensional feature vector 731 or 732 from each of the images 711 or 712. At this time, each of the N-dimensional feature vectors 731 or 732 includes an M-dimensional feature vector 7311 or 7321.

[0063] Here, DNN 721 is trained to extract M-dimensional feature vector 7311 as first-type feature vector 41 and N-dimensional feature vector 731 as second-type feature vector 42. DNN 722 is trained to extract M-dimensional feature vector 7321 as first-type feature vector 41 and N-dimensional feature vector 732 as second-type feature vector 42.

[0064] The first loss function 741 receives M-dimensional feature vectors 7311 and 7321 as input and is designed to improve the discrimination performance of the first-form feature vector 41. The second loss function 742 receives N-dimensional feature vectors 731 and 732 as input and is designed to improve the discrimination performance of the second-form feature vector 42. The weight update unit 74 updates the parameters (weights) of the DNN unit 72 based on the first loss function 741 and the second loss function 742.

[0065] For example, the first loss function 741 is a contrast loss function of the feature vectors 7311 and 7321. The contrast loss function L can be given by the following equation (11).

[0066]

number

[0067] In this case, to increase the recall, the margin is set low. The margin is called a hyperparameter and is adjusted based on experience or as learning progresses. D is the distance between the feature vectors 7311 and 7321, and for example, the Euclidean distance can be used.

[0068] The second loss function 742 may use the Contrastive Loss function as in the first loss function 741, or may use another function. When the Contrastive Loss function is used as the second loss function 742, the margin may be different from the value of the first loss function 741.

[0069] Although Euclidean distance may be used for D, other distance indices such as Manhattan distance may also be used, or similarity may be converted into distance for use. For example, when cosine similarity Sc is used, conversion may be performed as shown in the following equation (12) or (13).

[0070]

number

[0071]

number

[0072] After determining the first loss function 741 and the second loss function 742, the weight update unit 74 combines these to generate a loss function Ltotal. When the first loss function 741 is L1 and the second loss function 742 is L2, the loss function Ltotal can be given by the following equation (14).

[0073]

number

[0074] Here, s is a parameter in the range of 0≦s≦1, and does not need to be fixed throughout the learning period. For example, s may be constant throughout the entire learning period, or may be set to 1 or a value close to 1 at the beginning of learning and then gradually decreased to a predetermined value.

[0075] The weight update unit 74 updates the weight for the DNN unit 72 using the loss function Ltotal. Here, since the DNNs 721 and 722 configure a Siamese Network, the weights of both are updated simultaneously. The trained DNNs 721 and 722 can be used as the feature extraction unit 33 as they are.

[0076] As described above, according to the third embodiment, the learning device 7 trains the DNNs 721 and 722 using the first loss function 741 and the second loss function 742. This makes it possible to configure the feature extraction unit 33, using the DNN 721 or the DNN 722, to extract a feature vector 12A that has a high recall rate in the M-dimensional first format and high discrimination performance in the N-dimensional second format.

[0077] <Fourth embodiment> The fourth embodiment illustrates an example in which a DNN 721 or a DNN 722 trained by the learning device 7 in FIG. 7 is used as the feature extraction unit 33 in FIG.

[0078] FIG. 8 is a block diagram showing an example of the configuration of a discrimination device according to the fourth embodiment. 8, the classification device 8 includes a DNN 82, a feature amount storage unit 85, and a similarity calculation unit 86. The similarity calculation unit 86 includes a first similarity calculation unit 861 and a second similarity calculation unit 862.

[0079] The DNN 82 extracts an N-dimensional feature vector 832 from the image 81. At this time, the N-dimensional feature vector 832 includes an M-dimensional feature vector 831. Here, the DNN 82 extracts the M-dimensional feature vector 831 as a first-type feature vector 41, and extracts the N-dimensional feature vector 732 as a second-type feature vector 42.

[0080] The feature amount storage unit 85 stores N-dimensional feature vectors 852 extracted in advance from images of a large number of known people. At this time, the N-dimensional feature vectors 852 include an M-dimensional feature vector 851. Here, the M-dimensional feature vectors 851 are extracted as the first-type feature vectors 41, and the N-dimensional feature vectors 852 are extracted as the second-type feature vectors 42.

[0081] The first similarity calculation unit 861 compares the M-dimensional feature vector 831 output from the DNN 82 with the M-dimensional feature vector 841 read from the feature storage unit 85, and calculates a first similarity between the feature vectors 831 and 841. The second similarity calculation unit 862 compares the N-dimensional feature vector 832 output from the DNN 82 with the N-dimensional feature vector 842 read from the feature storage unit 85, and calculates a second similarity between the feature vectors 832 and 842.

[0082] At this time, the second similarity calculation unit 862 compares, among the N-dimensional feature vectors 842 read from the feature storage unit 85, N-dimensional feature vectors 842 including M-dimensional feature vectors 841 whose first similarity is greater than a predetermined value with the N-dimensional feature vector 832. Therefore, the second similarity calculation unit 862 can omit comparing, with the N-dimensional feature vector 832, N-dimensional feature vectors 842 including M-dimensional feature vectors 841 whose first similarity is less than or equal to the predetermined value. Generally, most of the N-dimensional feature vectors 852 stored in the feature storage unit 85 are feature vectors of images of other people, so the first similarity between most of the N-dimensional feature vectors 852 and the M-dimensional feature vector 841 is less than or equal to the predetermined value. Therefore, even when a large number of N-dimensional feature vectors 852 are stored in the feature storage unit 85, the second similarity calculation unit 862 can complete the comparison with the N-dimensional feature vector 832 in a short time.

[0083] Fifth Embodiment Fig. 9 is a diagram showing an example of a model used for feature extraction according to the fifth embodiment. In Fig. 9, a neural network is shown as an example of a model used for feature extraction. The model in Fig. 9 can be used as the DNN 82 in Fig. 8.

[0084] 9, the neural network includes an input layer LI, a hidden layer LH, and an output layer LO. The input layer LI includes a node NI, the hidden layer LH includes a node NH, and the output layer LO includes a node NO. The node NI is connected to the node NH via an edge EI, the nodes NH are connected to each other via an edge EH, and the node NH is connected to the node NO via an edge EO.

[0085] The input layer LI receives input data DI including an object to be identified. The output layer LO outputs an N-dimensional second-type feature vector 42 including an M-dimensional first-type feature vector 41 for the object to be identified input to the input layer LI. The intermediate layer LH receives the output of the input layer LI as input, and is trained so that an N-dimensional second-type feature vector 42 including the M-dimensional first-type feature vector 41 for the object to be identified input to the input layer LI is output via the output layer LO. At this time, in the neural network training stage, a trained model can be generated by training the neural network using the training device 7 of FIG. 7.

[0086] In the inference stage of the neural network, input data DI including an object to be classified is input to the input layer LI. Then, the neural network performs operations on the output of the input layer LI in the hidden layer LH, and outputs an N-dimensional second-type feature vector 42 including an M-dimensional first-type feature vector 41 for the object to be classified from the output layer LO.

[0087] At this time, data input to each node NI in the input layer LI is weighted by the weight of each edge EI, and then input to each node NH in the hidden layer LH, and the inputs to each node NH are added together for each node NH to become the value of each node NH. The value of each node NH in the hidden layer LH is nonlinearly transformed using an activation function (e.g., a sigmoid function). Then, the transformed value is weighted by the weight of each edge EH, EO, and then input to each node NO in the output layer LO, and the inputs to each node NO are added together for each node NO to become the value of each node NO.

[0088] Sixth Embodiment Fig. 10 is a block diagram showing an example of the hardware configuration of an identification device according to the sixth embodiment. Note that this identification device 100 can provide the functions realized by software among the functional blocks shown in Figs. 1, 2, and 8.

[0089] 10, the identification device 100 includes a processor 101, a communication control device 102, a communication interface 103, a primary storage device 104, an auxiliary storage device 105, and an input / output interface 107. The processor 101, the communication control device 102, the communication interface 103, the primary storage device 104, the auxiliary storage device 105, and the input / output interface 107 are interconnected via an internal bus 106. The primary storage device 104 and the auxiliary storage device 105 are accessible from the processor 101.

[0090] In addition, an input device 111, an output device 112, a camera 113, and a gate 114 are provided outside the identification device 100. The input device 111, the output device 112, the camera 113, and the gate 114 are connected to the internal bus 106 via an input / output interface 107. The input device 111 and the output device 112 can be used as a human interface.

[0091] The input device 111 is, for example, a keyboard, a mouse, a touch panel, a card reader, an audio input device, etc. The output device 112 is, for example, a screen display device (liquid crystal monitor, organic EL (Electro Luminescence) display, graphic card, etc.), an audio output device (speaker, etc.), a printer, etc. The camera 113 captures, for example, an image of a person's face, etc. The gate 114 opens and closes based on the person identification result by the identification device 100. The gate 114 is, for example, an entrance / exit gate of an entrance / exit management system.

[0092] The processor 101 is hardware that controls the overall operation of the identification device 100. The processor 101 may be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The processor 101 may be a single-core processor or a multi-core processor. The processor 101 may include a hardware circuit (for example, an FPGA or an ASIC) such as an accelerator that performs part of the processing. The processor 101 may operate as a neural network.

[0093] The primary storage device 104 can be configured, for example, from a semiconductor memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The primary storage device 104 can store programs being executed by the processor 101 and can provide a work area for the processor 101 to execute programs.

[0094] The auxiliary storage device 105 is a storage device with a large storage capacity, such as a hard disk drive or a solid-state drive (SSD). The auxiliary storage device 105 can hold executable files for various programs and data used to execute the programs. The auxiliary storage device 105 can store a classification program 105A, a feature vector 105B, a learning program 105C, and a model 105D. The classification program 105A, the learning program 105C, and the model 105D may be software that can be installed in the classification device 100, or may be embedded in the classification device 100 as firmware. The feature vector 105B is an N-dimensional second-type feature vector 42 that includes an M-dimensional first-type feature vector 41 extracted from images of a large number of known people.

[0095] The communication control device 102 is hardware having a function of controlling communication with the outside. The communication control device 102 is connected to a network 109 via a communication interface 103. The network 109 may be the Internet or a Wide Area Network (WAN), or a Local Area Network (LAN) such as Wi-Fi or Ethernet (registered trademark). The network 109 may be a mixture of the Internet, a WAN, and a LAN.

[0096] The input / output interface 107 converts data input from the input device 111 and the camera 113 into a data format that can be processed by the processor 101. The input / output interface 107 also converts data output from the processor 101 into a data format that can be processed by the output device 112 and the gate 114.

[0097] The processor 101 can implement the processing of Fig. 3, Fig. 5, or Fig. 6 by reading the identification program 105A and the feature vector 105B stored in the auxiliary storage device 105 into the primary storage device 104 and executing them. At this time, the identification program 105A extracts a feature vector from a face image captured by the camera 113. The identification program 105A can then implement face authentication by comparing the feature vector extracted from the face image captured by the camera 113 with the feature vector 105B read from the auxiliary storage device 105. Here, the identification program 105A can call the model 105D to extract the feature vector from the face image captured by the camera 113, and receive the feature vector extracted from the face image captured by the camera 113 from the model 105D.

[0098] Furthermore, processor 101 can read learning program 105C stored in auxiliary storage device 105 into primary storage device 104 and execute it to cause model 105D to extract a feature vector from a face image captured by camera 113. Here, learning program 105C causes model 105D to learn so that an N-dimensional second-type feature vector 42 including an M-dimensional first-type feature vector 41 of FIG. 4 is extracted from a face image captured by camera 113. In this case, learning program 105C can realize the function of learning device 7 of FIG. 7.

[0099] Note that execution of the program for realizing the processing of Fig. 3, Fig. 5, or Fig. 6 may be shared among multiple processors or computers. Alternatively, processor 101 may instruct a cloud computer or the like via network 109 to execute all or part of the program for realizing the processing of Fig. 3, Fig. 5, or Fig. 6, and receive the execution results.

[0100] <Other embodiments> In the present invention, a program that realizes one or more functions of the above-described embodiments may be supplied to a system or device via a network or a storage medium. One or more functions of the above-described embodiments may also be realized by one or more processors in a computer of the system or device reading and executing the program. Alternatively, one or more functions may also be realized by a circuit (e.g., an FPGA or ASIC) that realizes one or more functions. While preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the spirit thereof. [Explanation of symbols]

[0101] 30 Identification device, 31 Image input unit, 32 Detection unit, 33 Feature extraction unit, 34 Identification unit, 340 Matching unit, 341 First similarity calculation unit, 342 Second similarity calculation unit, 35 Feature storage unit

Claims

1. extraction means for extracting an N-dimensional (N is an integer greater than M) second-type feature vector including an M-dimensional (M is an integer greater than 2) first-type feature vector from input data; a first calculation means for calculating a first similarity between the feature vectors of the first format; second calculation means for calculating a second similarity between the feature vectors of the second format when the first similarity is greater than a predetermined value; a classification means for classifying a classification target of the input data based on the feature vector of the first format and the feature vector of the second format; Equipped with A classification device, wherein the first similarity has a higher recall rate than the second similarity.

2. 2. The identification device according to claim 1, wherein the input data is image data, and the identification target is an object.

3. 3. The identification device according to claim 1, wherein the second similarity has a higher degree of conformance than the first similarity.

4. 4. The identification device according to claim 1, wherein the second calculation means calculates the second similarity based on an intermediate result when the first calculation means calculates the first similarity.

5. 5. The classification device according to claim 1, wherein the extraction means is a deep neural network trained using a first loss function designed to increase recall using the feature vector of the first format and a second loss function designed to increase classification performance using the feature vector of the second format.

6. extracting an N-dimensional (N is an integer greater than M) second-type feature vector from input data, the N-dimensional (N is an integer greater than M) second-type feature vector including an M-dimensional (M is an integer greater than or equal to 2) first-type feature vector; calculating a first similarity between the feature vectors of the first format; calculating a second similarity between the feature vectors of the second format when the first similarity is greater than a predetermined value; identifying a classification target of the input data based on the feature vector of the first format and the feature vector of the second format; wherein the second similarity has a higher degree of conformance than the first similarity.

7. A step of acquiring a set of data of the classification target together with a label indicating whether or not the data are the same classification target as training data; A learning method characterized by having a computer execute a step of training a deep neural network based on the training data, a first loss function designed to increase the recall rate using an M-dimensional (M is an integer greater than or equal to 2) first-type feature vector for the object to be identified, and a second loss function designed to increase the classification performance using an N-dimensional (N is an integer greater than M) second-type feature vector including the first-type feature vector for the object to be identified based on the training data.

8. A program for causing a computer to operate as the identification device according to any one of claims 1 to 5.

Images