Face recognition device and program therefor

The face recognition device automates the collection and metadata addition process, using large-scale datasets and web scraping to generate comprehensive learning data, enhancing accuracy and efficiency in face recognition.

JP2024164309AActive Publication Date: 2024-11-26NIPPON HOSO KYOKAI +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024153609
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-02-27
Filing Date
2024-09-06
Publication Date
2024-11-26
Estimated Expiration
2041-02-26

AI Technical Summary

Technical Problem

Existing face recognition technologies face inefficiencies due to manual labor in collecting facial images and the need for precise tuning of CG parameters, as well as the time-consuming process of manually adding metadata, and lack of association with category information for multiple angle recognition.

Method used

A face recognition device and program that automatically collects face images and adds metadata by analyzing large-scale datasets like Wikipedia, extracting keywords and category information, and using web scraping to associate facial features with person names and categories, without human intervention.

Benefits of technology

Enables accurate and efficient face recognition by automatically generating comprehensive learning data with metadata, allowing for high-accuracy face recognition across various angles and categories.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164309000001_ABST
    Figure 2024164309000001_ABST
Patent Text Reader

Abstract

To provide a face recognition device for recognizing the face of a person seen in a video.SOLUTION: A face recognition device 2 includes face detection means 21 for detecting a face seen in a video and extracting a face image, feature amount extraction means 22 for extracting a feature amount from the face image, similarity calculation means 23 for calculating similarity between a feature amount of learning data for face recognition corresponding to designated category information and the feature amount extracted by the feature amount extraction means 22 in the learning data for face recognition obtained by associating the feature amount of the face image and the category information with a person name, and person specification means 24 for making a person name corresponding to the feature amount of the learning data for face recognition whose similarity is larger than a predetermined threshold a recognition result.SELECTED DRAWING: Figure 11
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a face recognition device and a program therefor. [Background technology]

[0002] In recent years, advances in AI technology have made it possible to identify people by recognizing their faces in images and videos with high accuracy, and this has led to practical applications in the security field. In addition, efforts to use face recognition technology to add metadata such as keywords to videos are becoming more common, primarily among broadcasting stations. These facial recognition technologies are based on a mechanism in which face images for learning are collected, facial features that can recognize differences between people are calculated through machine learning processing, and faces are recognized based on the similarity of those facial features (see Patent Document 1). The performance of this face recognition process is greatly affected by the quantity and quality of the training data used in the training process. The method of generating this training data is generally to collect face images through many manual processes.

[0003] On the other hand, as a method for collecting face images, a technology has been disclosed in which a CG model of a 3D face image is generated from collected 2D face images, and a larger number of face images are generated by manipulating CG parameters (see Patent Document 2). However, this method requires precise tuning of the CG parameters in order to generate a CG model of an appropriate face image. In addition, a commonly used practical method is to collect facial images using web scraping technology. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-33372 A [Patent Document 2] Special Publication No. 2016-501396 Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, most of the methods for collecting face images involve manual labor, which is problematic. Even if a method for generating face images is used, such as the invention described in Patent Document 2, the generation method requires precise tuning of CG parameters, and the problem of laborious labor remains. Furthermore, the conventional method is a method for collecting face images, and in order to use the collected face images as learning data for face recognition, metadata such as person names must be added to the images.

[0006] However, with conventional methods, adding such metadata must be done manually after collecting face images, which is time-consuming and inefficient. Furthermore, in face recognition technology, in order to utilize face recognition results from multiple angles, it is desirable to associate a face image with category information to which the person belongs.

[0007] The present invention has been made in consideration of such problems, and has an objective of providing a face recognition device and a program therefor that can collect face images without manual effort and recognize the faces of people appearing in a video from learning data for face recognition to which metadata has been added. [Means for solving the problem]

[0008] In order to solve the above problems, the face recognition device of the present invention is a face recognition device that recognizes the faces of people in a video using face recognition learning data in which person names are associated with facial image features and category information, and is configured to include a face detection means, a feature extraction means, a similarity calculation means, and a person identification means.

[0009] In such a configuration, the face recognition device detects a face captured in the video by the face detection means and extracts a face image. Then, the face recognition device extracts features from the face image using the feature extraction means. Furthermore, the face recognition device calculates, by a similarity calculation means, the similarity between the features of the face recognition learning data corresponding to the specified category information among the face recognition learning data and the features extracted by the feature extraction means. Then, the face recognition device uses the person identification means to determine as a recognition result the name of a person corresponding to a feature amount of the face recognition learning data whose similarity is greater than a predetermined threshold value. This face recognition device can be operated by a face recognition program that causes a computer to function as each of the means described above. Effect of the Invention

[0010] The present invention provides the following excellent effects. According to the face recognition device of the present invention, face recognition can be performed by specifying a category, so that highly accurate face recognition can be performed. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a face recognition learning data generating device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a data structure diagram of a keyword table showing an example of the correspondence between IDs (identifiers) and keywords in the data structure of a data set. [Diagram 3] FIG. 2 is a data structure diagram of a category table showing an example of a correspondence relationship between IDs (identifiers) and category IDs in the data structure of a data set. [Figure 4] FIG. 2 is a data structure diagram of a hierarchical table showing an example of the correspondence between IDs (identifiers) and upper hierarchical IDs in the data structure of a data set. [Diagram 5] FIG. 2 is an explanatory diagram for explaining an example of a hierarchical structure of categories. [Figure 6] FIG. 11 is an explanatory diagram for explaining the relationship between person KWs and category information. [Figure 7] FIG. 1 is a network diagram showing an example of the configuration of a neural network model that extracts facial features from a facial image. [Figure 8] FIG. 1 is an explanatory diagram for explaining a concept of clustering of facial feature amounts. [Figure 9] 1 is a data structure diagram showing an example of a structure of learning data for face recognition. [Figure 10] 4 is a flowchart showing the operation of the face recognition learning data generating device according to the embodiment of the present invention. [Figure 11] 1 is a block diagram showing a configuration of a face recognition device according to an embodiment of the present invention. [Figure 12] 4 is a flowchart showing the operation of the face recognition device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Configuration of a learning data generator for face recognition] First, with reference to FIG. 1, a configuration of a face recognition learning data generating device 1 according to an embodiment of the present invention will be described. The face recognition learning data generating device 1 generates face recognition learning data. As shown in FIG. 1, the face recognition learning data generating device 1 includes a data analysis means 10, a face image collection means 11, a person keyword correspondence determination means 12, and a data integration means 13.

[0013] The data analysis means 10 analyzes the data set and extracts person name keywords (hereinafter, person KWs) and keywords for categories (attributes) to which the person belongs (hereinafter, category KWs) from keywords containing person names.

[0014] The dataset is a large-scale data set that associates keywords, including people's names, with hierarchical categories. Any dataset may be used as long as the data structure is known, but here, we will use Wikipedia data as an example. First, an example of the data structure of a data set will be described with reference to FIGS.

[0015] FIG. 2 is a data structure diagram of a keyword table Tk showing an example of the correspondence between IDs (identifiers) and keywords in the data structure of a data set. An ID is an identifier such as a number that is individually assigned to a keyword. A keyword is a character string of various headwords including a person's name, and is a Wikipedia page headword. In the example of FIG. 2, the keyword "Japanese politician" is associated with the ID "id1", the keyword "XXXX Taro" is associated with the ID "id6", etc. This allows various keywords, including people's names, to be uniquely identified by the ID.

[0016] FIG. 3 is a data structure diagram of a category table Tc showing an example of the correspondence between IDs (identifiers) and category IDs in the data structure of a dataset. A category ID is a list of IDs of categories to which keywords belong. In the example of FIG. 3, ID "id6" is associated with IDs "id1, id2, id3, id4, ..., id8, ..." as category IDs. This means that the keyword "〇〇Taro" corresponding to ID "id6" belongs to the keyword (category KW) "Japanese politicians" corresponding to ID "id1", the keyword (category KW) "politicians of each prefecture" corresponding to ID "id2", etc. This identifies which category the keyword belongs to.

[0017] FIG. 4 is a data structure diagram of a hierarchical table Tu showing an example of the correspondence between IDs (identifiers) of category keywords and higher-level IDs in the data structure of a dataset. The higher-level IDs are a list of IDs in a higher level in a hierarchical structure for a certain ID. In the example of FIG. 4, IDs "id1, id2, id3" are associated with ID "id4" as higher-level IDs. This means that the keyword "Japanese politician" corresponding to ID "id1", the keyword "politicians of each prefecture" corresponding to ID "id2", and the keyword "Tokyo politician" corresponding to ID "id3" are in a higher level as categories for the category keyword "Diet members elected from Tokyo" corresponding to ID "id4". This allows the keywords to be organized into a hierarchy. When the data set is Wikipedia data, there is also a table in which an explanation of a keyword is associated with an ID, but this is not shown here.

[0018] Returning to FIG. 1, the configuration of the face recognition learning data generation device 1 will be further described. The data analysis means 10 includes a category hierarchy means 100, a person KW extraction means 101, and a category information extraction means .

[0019] The category hierarchical division means 100 generates category hierarchical information by associating category IDs with position information in which categories are hierarchically divided, from a data set. Specifically, the category hierarchical division means 100 refers to the hierarchy table Tu in Fig. 4, and sequentially and recursively associates each ID (identifier) ​​with a higher hierarchy ID, thereby generating a tree structure graph as shown in Fig. 5. Here, an example of a tree structure in which the category KW "Japanese politicians" is the highest hierarchy will be described. In addition, in Wikipedia, the hierarchy of categories is structured as a directed acyclic type, and since there is a possibility that the same category keyword may appear in the recursive process, the maximum number of recursions is set in advance and the process is stopped.

[0020] Then, the category hierarchical unit 100 generates category hierarchical information by associating the ID of the category keyword with the position information of each category keyword configured in the tree-structured graph. This position information is information indicating the position of a node (corresponding to a category keyword) in the tree-structured graph, and can be expressed in, for example, a tuple format. In the tuple format of the position information, each number in the order of the numeric string indicates the position of a node to be traced for each layer of the tree-structured graph.

[0021] For example, suppose the position information of a node is (0,2,25,11,2). Each number indicates the position of the child node from the left of the parent node, starting from the top layer. The leftmost value in the position information is '0', as it indicates the topmost (first layer) node (root node). The second number in the position information, '2', indicates the second child node from the left of the parent node, one layer below the root node (second layer). For example, in FIG. 5, the location information of the category keyword "Japanese politicians" for id1 is (0), and the location information of the category keyword "Tokyo politicians" for id3 is (0,1,1). The category hierarchical division means 100 outputs the generated category hierarchical information (ID and location information) to the category information extraction means 102.

[0022] The person KW extraction means (person keyword extraction means) 101 extracts person name keywords (person KWs) from the data set. Here, the person KW extraction means 101 extracts keywords corresponding to categories including predetermined character strings that identify a person from the data set as person KWs. Specifically, the person KW extraction means 101 searches for keywords that include predetermined words that can be used to identify a person, such as "year of birth," "year of death," and "living person," and extracts keywords that include the ID corresponding to the keyword as a category ID as person KWs.

[0023] For example, the person KW extraction means 101 searches for IDs containing the keyword phrase "grade" in the keyword table Tk of Fig. 2. Then, the person KW extraction means 101 further searches for IDs containing the searched ID (id8 in this case) as a category ID in the category table Tc of Fig. 3. Then, the person KW extraction means 101 extracts "XXXX Taro" corresponding to the searched ID (id6 in this case) as a person KW in the keyword table Tk of Fig. 2. In addition, the process of identifying keywords that include in the categories of "birth year," "death," or "living person" in Wikipedia as person KWs has been experimentally confirmed to be 100% accurate.

[0024] Of course, as a keyword for a person in a data set, a keyword containing a phrase that identifies the person in the person's name may be used. For example, a keyword containing a predetermined phrase "person" such as "XXXX Taro (person)" may be used. In this case, the person KW extraction means 101 may extract a keyword containing the phrase "person" as a person KW in the keyword table Tk. The person KW extraction means 101 outputs the extracted person KW to the category information extraction means 102 and the face image collection means 11.

[0025] The category information extraction means 102 extracts, as category information, hierarchical categories corresponding to keywords of person names from the data set, based on the category hierarchical information hierarchized by the category hierarchization means 100 . The category information extraction means 102 extracts upper IDs in the tree structure from the category hierarchical information using as a reference the category ID corresponding to the person KW extracted by the person KW extraction means 101. Then, the category information extraction means 102 generates keywords corresponding to the reference category ID and the upper category ID as category information.

[0026] For example, even if only "id5" is set as the category ID for a certain person keyword "△△Hanako" in the category table Tc (Figure 3), the category information extraction means 102 will generate, as category information, keywords corresponding to the ID "id5" and the IDs "id1, id2, id3, id4" that are hierarchically ranked higher in the category hierarchy information, as shown in Figure 6. This allows the category information extraction means 102 to associate not only the category KW that directly corresponds to the person KW, but also the category KW that is higher in the hierarchical structure with the person KW. The category information extraction means 102 associates the person KW with the category information and outputs it to the data integration means 13.

[0027] Note that each keyword in the category information may be added with position information in a tree structure, which makes it possible to construct a database by hierarchically organizing the learning data for face recognition to be generated.

[0028] The face image collecting means 11 collects face images corresponding to the person KW. The face image collection means 11 includes a web search means 110 and a face image extraction means 111 .

[0029] The Web search means 110 searches for and collects documents (Web documents) related to keywords of person names on the Web, which is a system for publishing and viewing documents on a network (Internet) N. The Web search means 110 searches and collects Web documents related to the person KWs extracted by the person KW extraction means 101 of the data analysis means 10 from the Web server S by a general search engine via the network N. Note that the search for person KWs by the search engine may use a general Web scraping technique. The Web documents collected by the Web search means 110 include images including the face of the person KW, images other than the face, text data, and the like. The Web search means 110 outputs the collected Web documents to the face image extraction means 111 .

[0030] The facial image extraction means 111 extracts facial images from images contained in Web documents collected by the Web search means 110 by facial recognition. The face image extraction means 111 detects faces in images contained in Web documents by general face recognition processing, and extracts only images in which one face appears. The face image extraction means 111 outputs the extracted face image and the Web document from which the face image was extracted, together with the person KW, to the person KW correspondence determination means 12.

[0031] The person keyword correspondence determination means (person keyword correspondence determination means) 12 determines whether or not the face image collected by the face image collection means 11 is a face image of a person of the person keyword. The person KW correspondence determination means 12 includes a feature amount extraction means 120 , a clustering means 121 , a relevance setting means 122 , and a determination means 123 .

[0032] The feature extraction means 120 extracts features (facial feature amounts) from the face images collected by the face image collection means 11 . For example, the feature extraction means 120 extracts facial feature F by performing multiple convolution operations on a facial image I using a pre-trained convolutional neural network NN equipped with multiple convolution layers CL (CL1, CL2, ...) as shown in Fig. 7. This facial feature F has a dimension of, for example, about 500. This convolutional neural network NN can use, for example, a network for the feature extraction part of a general object recognition network such as VGG (Visual Geometry Group) 19. The feature extraction means 120 extracts face features {f1, f2, . . . , f n} (n is the number of face images) for each person KW to the clustering means 121.

[0033] The clustering means 121 clusters the facial features extracted by the feature extraction means 120 . A general method may be used as the clustering method in the clustering means 121. For example, the clustering means 121 uses a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, which can also calculate the number of classes, to calculate the facial feature amount {f1, f2, ..., f n} are clustered.

[0034] Fig. 8 shows the concept of face features clustered by the clustering means 121. Fig. 8 shows a state in which face features {f1, f2, ..., f9} are classified into three classes (C1, C2, C3), with face features {f1, f4, f5, f9} classified into class C1, face features {f6, f8} classified into class C2, and face features {f2, f3, f7} classified into class C3. This enables the clustering means 121 to classify a plurality of facial features into a plurality of groups, such as a group of people corresponding to the person KW and other groups, or even if the people correspond to the same person KW, into groups of faces facing forward and faces facing sideways, etc. The clustering means 121 outputs the clustering result to the determination means 123 .

[0035] The relevance setting means 122 sets a relevance indicating the degree of relevance between the facial feature amount and a person's name in association with the facial feature amount based on a Web document including a face image from which the facial feature amount has been extracted. The relevance setting means 122 sets the relevance based on a predetermined criterion. Here, the relevance setting means 122 increases the relevance based on the criterion from a predetermined initial value (e.g., "0").

[0036] For example, when the Web document from which the face image has been extracted is a Wikipedia document that uses a person keyword as an entry word, the relevance setting means 122 adds a predetermined number to the relevance. Also, for example, when a tag indicating an image in a Web document includes a description of a person keyword, the relevance setting means 122 adds a predetermined number to the relevance. Also, for example, when a Web document is similar to a description that describes a person keyword included in the data set, the relevance setting means 122 adds a predetermined number to the relevance. This similarity can be determined, for example, when the number of words common to the Web document and the description is greater than a predetermined number, or when the ratio of the number of common words to the total number of words in the Web document and the description is greater than a predetermined ratio. In addition, when Web documents are categorized in advance, such as in Wikipedia, it is possible to update the relevance by calculating the degree of overlap with the category information of the person keyword and adding it to the relevance. In the above relevance calculations, it is also possible to obtain the relevance by vector converting the words and image features in the text, which are the elements for calculating the relevance, into a common dimensional space using WORD2VEC, PCA (Principal Component Analysis), etc., and then performing similarity calculation processing such as cosine similarity. The relevance setting means 122 sets relevance W{w1, w2, . . . , w n} (n is the number of face images) for each person KW to the determination means 123.

[0037] The determination means 123 determines whether or not each class corresponds to a person's name, based on the degree of association set for the face feature amounts belonging to each class clustered by the clustering means 121. This determination means 123 calculates the sum of the relevance set by the relevance setting means 122 for the facial features belonging to each clustered class, and determines that a class whose sum satisfies a predetermined criterion is the class corresponding to the person KW.

[0038] For example, suppose that the clustering result for a certain person KW is in the state shown in Fig. 8. Also, suppose that the relevance of the person KW set to the face feature amount {f1, f2, ..., f9} is {w1, w2, ..., w9}. In this case, the determination means 123 calculates the sum W1 of the relevance of class C1 as w1+w4+w5+w9, the sum W2 of the relevance of class C2 as w6+w8, and the sum W3 of the relevance of class C3 as w2+w3+w7.

[0039] In addition, the judgment means 123 judges whether each class corresponds to a person keyword based on, for example, the judgment criterion of whether the sum of the relevance degrees for each class exceeds a predetermined threshold, or whether the difference between the sum of the relevance degrees for each class and the sums of the other classes is greater than a predetermined threshold. The determination means 123 associates the person KW with the face feature amount determined to be the class of the person KW, and outputs the result to the data integration means 13.

[0040] The data integration means 13 integrates category information corresponding to the person keywords generated by the data analysis means 10 with facial features determined to correspond to the person keywords by the person keyword correspondence determination means 12, to generate learning data for face recognition. The data integration means 13 generates face recognition learning data by linking the face feature amount F and the category information CI for each person KW, as shown in Fig. 9. Note that position information in a tree structure may be added to each keyword in the category information CI. Furthermore, the data integration means 13 may add a face image from which facial features have been extracted to the face recognition learning data in place of or together with the facial features.

[0041] By configuring as described above, the face recognition learning data generation device 1 can collect face images and generate learning data for face recognition with added person KWs and category information without any manual effort. Furthermore, the face recognition learning data generation device 1 can collect face images in a wide variety of categories and also collect face images in hierarchical categories, thereby enabling the face recognition learning data generation device 1 to generate comprehensive and systematically organized learning data. The face recognition learning data generating device 1 can be operated by a program (face recognition learning data generating program) for causing a computer to function as each of the means described above.

[0042] [Operation of the face recognition learning data generation device] Next, the operation of the face recognition learning data generating device 1 according to the embodiment of the present invention will be described with reference to FIG. 10 (for the configuration, refer to FIG. 1 as appropriate). In step S1, the category hierarchical division means 100 of the data analysis means 10 analyzes the data set and divides categories into a hierarchy. Here, the category hierarchical division means 100 refers to a hierarchy table Tu (FIG. 4), sequentially and recursively associates a higher hierarchy ID with each ID (identifier), thereby dividing the categories into a hierarchy, and generates category hierarchy information (FIG. 5) in which the IDs and position information are associated with the nodes of a tree-structure graph.

[0043] In step S2, the person KW extraction means 101 of the data analysis means 10 analyzes the data set and extracts person KWs from the keywords of the data set. Here, the person KW extraction means 101 extracts, as person KWs, keywords in the keyword table Tk (FIG. 2) whose category table Tc (FIG. 3) contains an ID including any of the words "birth date," "death," and "living person."

[0044] In step S3, the category information extraction means 102 of the data analysis means 10 extracts category KWs corresponding to the person KWs extracted in step S2 from the dataset based on the category hierarchical information generated in step S1, and generates category information of the person KWs (Figure 6).

[0045] In step S4, the Web search means 110 of the face image collection means 11 collects Web documents corresponding to the person KW extracted in step S2 from the Web server S via the network N. Here, the Web search means 110 collects Web documents by Web scraping using the person KW.

[0046] In step S5, the facial image extraction means 111 of the facial image collection means 11 extracts facial images from images included in the Web documents collected in step S4. Here, the facial image extraction means 111 detects faces from the Web documents by facial recognition, and extracts only images in which one face appears.

[0047] In step S6, the feature extraction means 120 of the person KW correspondence determination means 12 extracts features (facial features) from the face image extracted in step S5. Here, the feature extraction means 120 extracts facial features from the face image using a pre-trained convolutional neural network. In step S7, the clustering means 121 of the person KW correspondence determination means 12 clusters the face feature amounts extracted in step S6.

[0048] In step S8, the relevance setting means 122 of the person KW correspondence determination means 12 sets the relevance between the facial feature extracted in step S6 and the person KW. For example, in a case where the Web document is a Wikipedia document with the person KW as an entry term, the relevance setting means 122 sets a larger relevance value as the relevance between the Web document and the person KW increases.

[0049] In step S9, the determination means 123 of the person KW correspondence determination means 12 determines, for each class clustered in step S7, whether or not the class corresponds to a person KW, based on the relevance set in step S8. Here, the determination means 123 calculates the sum of the relevance degrees for the facial features included in the class, and determines a class whose sum of the relevance degrees is greater than a predetermined threshold value as a class corresponding to a person KW.

[0050] In step S10, the data integration means 13 integrates, for each person KW, the category information generated in step S3 corresponding to the person KW and the facial features belonging to the class determined in step S9 to be the class of the person KW, to generate learning data for face recognition (FIG. 9). Through the above operations, the face recognition learning data generation device 1 can automatically collect face images corresponding to person KWs and generate face recognition learning data.

[0051] [Configuration of face recognition device] Next, the configuration of the face recognition device 2 according to the embodiment of the present invention will be described with reference to FIG. The face recognition device 2 recognizes the faces of people captured in an image. As shown in FIG. 11, the face recognition device 2 includes a face recognition learning data storage means 20, a face detection means 21, a feature amount extraction means 22, a similarity calculation means 23, and a person identification means 24.

[0052] The face recognition learning data storage means 20 stores face recognition learning data, and can be configured with a general storage medium such as a hard disk. In this face recognition learning data storage means 20, face recognition learning data generated by the face recognition learning data generating device 1 described with reference to FIG. 1 is stored in advance. The face recognition learning data is a database in which a person KW is associated with a face feature amount F and category information CI, as shown in FIG.

[0053] The face detection means 21 detects faces appearing in the input image. The face detection means 21 recognizes faces in the image by using a general face recognition process. The face detection means 21 extracts the detected face area (face image) from the video for each frame, and outputs it to the feature extraction means 22.

[0054] The feature extraction means 22 extracts facial features from the face image detected by the face detection means 21. The feature extraction means 22 extracts facial features using the same method as the feature extraction means 120 of the face recognition learning data generation device 1 described in FIG. The feature extraction means 22 outputs the extracted facial feature or features to the similarity calculation means 23 .

[0055] The similarity calculation means 23 calculates the degree of similarity (similarity) between the facial feature amount extracted by the feature amount extraction means 22 and the facial feature amount stored in the face recognition learning data storage means 20 . Here, the similarity calculation means 23 calculates the similarity for the facial feature amounts stored in the face recognition learning data storage means 20, which face feature amounts correspond to category information (keywords) designated from the outside. For example, when the keyword “politician in Tokyo” is specified as the category information, the similarity calculation means 23 calculates the similarity only for the facial features of the learning data that includes the keyword “politician in Tokyo” in the category information among the face recognition learning data stored in the face recognition learning data storage means 20.

[0056] The similarity calculation method used by the similarity calculation means 23 may be a general method, for example, the similarity can be calculated using the cosine similarity between the vector of facial features extracted by the feature extraction means 22 and the vector of facial features stored in the face recognition learning data storage means 20. The similarity calculation means 23 outputs the calculated similarity and the person KW of the face recognition learning data used to calculate the similarity to the person identification means 24.

[0057] The person identification means 24 identifies a person based on the similarity calculated by the similarity calculation means 23. The person identification means 24 extracts those whose similarities calculated by the similarity calculation means 23 are greater than a predetermined threshold. The person identification means 24 then sorts the extracted similarities and outputs the person KWs of the face recognition learning data used to calculate the similarities as recognition results in descending order of similarity. Of course, for simplicity, it is also possible to output only the person KW with the highest similarity.

[0058] In addition, the person identification means 24 may output a keyword of the specified category information in association with the person KW, or keywords of all category information associated with the person KW stored in the face recognition learning data storage means 20. This makes it possible to add metadata based on category information to a person KW recognized by the face recognition device 2. In addition, when tree-structured location information is added to the category information of the face recognition learning data stored in the face recognition learning data storage means 20, the person identification means 24 may output the location information in addition to the category information in association with the person KW.

[0059] With the above-described configuration, the face recognition device 2 can perform face recognition by specifying category information, thereby improving the accuracy of face recognition. The face recognition device 2 can be operated by a program (face recognition program) for causing a computer to function as each of the above-mentioned means.

[0060] [Facial recognition device in action] Next, the operation of the face recognition device 2 according to an embodiment of the present invention will be described with reference to Fig. 12 (for the configuration, refer to Fig. 11 as appropriate). It is assumed that the face recognition learning data storage means 20 stores in advance the face recognition learning data generated by the face recognition learning data generation device 1 (Fig. 1).

[0061] In step S20, the face detection means 21 detects a face appearing in the input video image. At this time, the face detection means 21 extracts the area of ​​the detected face as a face image. In step S21, the feature extraction means 22 extracts features (facial feature amounts) from the face area (facial image) detected in step S20.

[0062] In step S22, the similarity calculation means 23 calculates the degree of similarity (similarity) between the facial features extracted in step S21 and the facial features stored in the face recognition learning data storage means 20 and corresponding to externally specified category information (keywords). In step S23, the person identification means 24 identifies a person based on the similarity calculated in step S22. At this time, the person identification means 24 extracts and sorts those having a similarity greater than a predetermined threshold, and outputs the person KWs with the highest similarity as the recognition result. Through the above operations, the face recognition device 2 can limit the categories and recognize faces from the video.

[0063] Although the embodiment of the present invention has been described above, the present invention is not limited to this embodiment. In this embodiment, the dataset has keywords including a person's name, and the target person is identified using the keyword of the person's name. However, the keyword does not have to be a person's name as long as it can identify the target person. For example, the keyword may be a number representing a person. For example, it may be an index such as a My Number that can identify a person. Furthermore, the text of the dataset itself may have a person's name replaced with another index so that the individual cannot be identified.

[0064] Also, in this embodiment, a document including a face image is described as being collected by searching on the Web. However, the document including a face image is not limited to being collected by searching on the Web, and may be anything as long as it includes a face image and document data related to that face. For example, the document including a face image may be a document stored in a storage medium in advance, or may be a document read from an external source. Therefore, the Web search means 110 is not limited to searching on the Web, and may be configured as a collection means. The document here is a multimedia document including text, images, and audio.

[0065] Also, in this embodiment, the determination means 123 is described as calculating the sum of the relevance for each clustered class and making a determination based on the calculated value. However, the index used for the determination is not limited to the sum of the relevance for each class. The index used for the determination may be any index that can be determined to correspond to a person keyword. For example, the index may be the average relevance of the class, the maximum value of the facial feature amount belonging to the class, etc. [Explanation of symbols]

[0066] 1. Device for generating learning data for face recognition 10 Data Analysis Methods 100 Category Hierarchy Means 101 Person KW extraction method 102 Category information extraction method 11. Means of collecting face images 110 Web search methods (collection methods) 111 Facial image extraction means 12 Person keyword correspondence determination method 120 Feature Extraction Method 121 Clustering Methods 122 Means of setting relevance 123 Judgment means 13 Data Integration Methods 2. Facial Recognition Device 20. Facial recognition learning data storage means 21 Face detection means 22 Feature extraction method 23 Similarity calculation means 24 Person identification means

Claims

1. A face recognition device that recognizes the faces of people in a video using face recognition learning data in which face image features and category information are associated with person names, comprising: A face detection means for detecting a face shown in a video and extracting a face image; A feature extraction means for extracting features from the face image; a similarity calculation means for calculating a similarity between a feature of face recognition learning data corresponding to designated category information among the face recognition learning data and the feature extracted by the feature extraction means; a person identification means for identifying a person whose name corresponds to a feature of the face recognition learning data whose similarity is greater than a predetermined threshold value as a recognition result; A face recognition device comprising:

2. A face recognition program for causing a computer to function as the face recognition device according to claim 1.

Citation Information

Patent Citations

  • Human body collation device

    JP1999283030A

  • Image authentication device, image authentication method, computer program and storage medium

    JP2019185384A

  • How to generate a 3D face model

    JP2016501396A

  • Person recognition device and program therefor

    JP2017033372A