Facial recognition device and its program

Abstract

To provide a face recognition device for recognizing the face of a person seen in a video.SOLUTION: A face recognition device 2 includes face detection means 21 for detecting a face seen in a video and extracting a face image, feature amount extraction means 22 for extracting a feature amount from the face image, similarity calculation means 23 for calculating similarity between a feature amount of learning data for face recognition corresponding to designated category information and the feature amount extracted by the feature amount extraction means 22 in the learning data for face recognition obtained by associating the feature amount of the face image and the category information with a person name, and person specification means 24 for making a person name corresponding to the feature amount of the learning data for face recognition whose similarity is larger than a predetermined threshold a recognition result.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a face recognition device and a program therefor. [Background technology]

[0002] In recent years, advances in AI technology have made it possible to identify people by recognizing their faces in images and videos, and this has led to practical applications in the security field. Furthermore, efforts to use facial recognition technology to add metadata such as keywords to videos are gaining momentum, particularly among broadcasters. These facial recognition technologies are based on a mechanism in which face images for learning are collected, facial features that can recognize differences between people are calculated through machine learning processing, and faces are recognized based on the similarity of those facial features (see Patent Document 1). The performance of this facial recognition process is greatly affected by the quantity and quality of the training data used in the training process. The method of generating this training data is usually to manually collect and generate facial images.

[0003] On the other hand, as a method for collecting face images, a technology has been disclosed in which a CG model of a 3D face image is generated from collected 2D face images and a larger number of face images are generated by manipulating the CG parameters (see Patent Document 2). However, this method requires precise tuning of the CG parameters in order to generate a CG model of an appropriate face image. Another commonly used method in practice is to collect facial images using web scraping technology. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-33372 [Patent Document 2] Special Publication No. 2016-501396 Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, most of the methods for collecting face images involve manual labor, which is a problem. Even if a method for generating face images is used, such as the invention described in Patent Document 2, the generation method requires precise tuning of CG parameters, which still leaves the problem of being time-consuming. Furthermore, conventional methods are methods for collecting face images, and in order to use the collected face images as learning data for face recognition, it is necessary to add metadata such as person names to the collected face images.

[0006] However, conventional methods have the problem that adding such metadata must be done manually after collecting face images, which is time-consuming and inefficient. Furthermore, in face recognition technology, in order to utilize face recognition results from multiple angles, it is desirable that face images be associated with category information to which the person belongs.

[0007] The present invention has been made in consideration of such problems, and aims to provide a face recognition device and a program therefor that can collect face images without manual effort and recognize the faces of people appearing in a video from face recognition learning data to which metadata has been added. [Means for solving the problem]

[0008] In order to achieve the above object, a face recognition device according to the present invention includes: person keyword extraction means for extracting keywords of person names from a data set in which keywords are associated with hierarchical categories; category information extraction means for extracting, from the data set, the hierarchical categories corresponding to the keywords that identify a person as category information; collection means for collecting documents related to the keywords that identify the person; face image extraction means for extracting face images from images included in the documents by face recognition; feature extraction means for extracting features from the face images; association setting means for setting associations between the features and a person identified by the keywords and the features based on the documents including the face images from which the features have been extracted; clustering means for clustering the features of the face images; determination means for determining whether each of the classes corresponds to the identified person based on the associations set for the features belonging to each clustered class; and data integration means for generating face recognition learning data by integrating, for each keyword that identifies the person, the features belonging to the class determined to correspond to the identified person and the category information. a face recognition learning data storage means for storing the face recognition learning data; and a face recognition device for detecting faces in an image and extracting a face image. A face detection means; Extracting features from the face image extracted by the face detection means second feature extraction means; Calculating the similarity between the feature amount of a face image corresponding to designated category information in the face recognition learning data and the feature amount extracted by the second feature amount extraction means a similarity calculation means; The person name corresponding to the feature amount whose similarity is greater than a predetermined threshold value is determined as the recognition result. and a person identification means.

[0009] face The recognition device can be operated by a face recognition program that causes a computer to function as each of the above-mentioned means. [Effects of the Invention]

[0010] The present invention provides the following excellent effects. According to the face recognition device of the present invention, face recognition can be performed by specifying a category, so that highly accurate face recognition can be performed. [Brief explanation of the drawings]

[0011] [Figure 1]1 is a block diagram showing the configuration of a face recognition learning data generation device according to an embodiment of the present invention; [Figure 2] FIG. 10 is a data structure diagram of a keyword table showing an example of the correspondence between IDs (identifiers) and keywords in the data structure of a data set. [Figure 3] FIG. 2 is a data structure diagram of a category table showing an example of the correspondence between IDs (identifiers) and category IDs in the data structure of a data set. [Figure 4] FIG. 10 is a data structure diagram of a hierarchical table showing an example of the correspondence between IDs (identifiers) and upper hierarchical IDs in the data structure of a data set. [Figure 5] FIG. 2 is an explanatory diagram illustrating an example of a hierarchical structure of categories. [Figure 6] FIG. 10 is an explanatory diagram for explaining the relationship between person KWs and category information. [Figure 7] FIG. 1 is a network diagram showing an example of the configuration of a neural network model that extracts facial features from a facial image. [Figure 8] FIG. 10 is an explanatory diagram illustrating the concept of clustering of facial feature amounts. [Figure 9] FIG. 2 is a data structure diagram showing an example of the structure of face recognition learning data. [Figure 10] 4 is a flowchart showing the operation of the face recognition learning data generation device according to the embodiment of the present invention. [Figure 11] 1 is a block diagram showing the configuration of a face recognition device according to an embodiment of the present invention. [Figure 12] 4 is a flowchart illustrating an operation of the face recognition device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Configuration of a face recognition learning data generation device] First, with reference to FIG. 1, the configuration of a face recognition learning data generation device 1 according to an embodiment of the present invention will be described. The face recognition learning data generation device 1 generates face recognition learning data. As shown in FIG. 1, the face recognition learning data generation device 1 includes a data analysis means 10, a face image collection means 11, a person keyword correspondence determination means 12, and a data integration means 13.

[0013] The data analysis means 10 analyzes the data set and extracts keywords for person names (hereinafter referred to as person KWs) and keywords for categories (attributes) to which the person belongs (hereinafter referred to as category KWs) from keywords containing person names.

[0014] The dataset is a large-scale data set that associates keywords, including people's names, with hierarchical categories. Any dataset can be used as long as the data structure is known, but here we will use Wikipedia data as an example. First, an example of the data structure of a data set will be described with reference to FIGS.

[0015] Figure 2 is a data structure diagram of the keyword table Tk, which shows an example of the correspondence between IDs (identifiers) and keywords in the data structure of the dataset. An ID is an identifier, such as a number, that is individually assigned to a keyword. A keyword is a string of various headwords, including people's names, and is the headword of a Wikipedia page. In the example of Figure 2, the keyword "Japanese politician" is associated with the ID "id1", the keyword "Taro X", and so on. This allows various keywords, including people's names, to be uniquely identified by the ID.

[0016] FIG. 3 is a data structure diagram of a category table Tc showing an example of the correspondence between IDs (identifiers) and category IDs in the data structure of a dataset. A category ID is a list of IDs of categories to which keywords belong. In the example of FIG. 3, ID "id6" is associated with IDs "id1, id2, id3, id4, ..., id8, ..." as category IDs. This means that the keyword "Taro X" corresponding to ID "id6" belongs to the keyword (category KW) "Japanese politicians" corresponding to ID "id1", the keyword (category KW) "politicians of each prefecture" corresponding to ID "id2", etc. This identifies which category the keyword belongs to.

[0017] FIG. 4 is a data structure diagram of a hierarchy table Tu, which shows an example of the correspondence between IDs (identifiers) of category keywords and upper-level IDs in the data structure of a dataset. Upper-level IDs are a hierarchical list of IDs in the upper level for a certain ID. In the example of FIG. 4, IDs "id1, id2, id3" are associated with ID "id4" as upper-level IDs. This means that for the category keyword "Diet members elected from Tokyo" corresponding to ID "id4," the keyword "Japanese politicians" corresponding to ID "id1," the keyword "politicians from each prefecture" corresponding to ID "id2," and the keyword "Tokyo politicians" corresponding to ID "id3" are in the upper level as categories. This allows the keywords to be organized into a hierarchy. If the dataset is Wikipedia data, there will be a table in which the IDs are associated with explanatory text that explains the keywords, but this will not be shown here.

[0018] Returning to FIG. 1, the configuration of the face recognition learning data generation device 1 will be further described. The data analysis means 10 includes a category hierarchy means 100, a person KW extraction means 101, and a category information extraction means .

[0019] The category layering means 100 generates category layer information by associating category IDs with position information representing layered categories from a data set. Specifically, the category layering means 100 refers to the layer table Tu in Fig. 4 and sequentially associates each ID (identifier) ​​with a higher layer ID in a recursive manner, thereby generating a tree structure graph as shown in Fig. 5. Here, an example of a tree structure in which the category keyword "Japanese politicians" is at the top layer will be described. In addition, in Wikipedia, the hierarchy of categories is a directed acyclic structure, and since there is a possibility that the same category keyword will appear during recursive processing, the maximum number of recursions will be set in advance and the processing will be stopped.

[0020] Then, the category layering means 100 generates category layer information by associating the IDs of the category keywords with the position information of each category keyword configured in the tree-structured graph. This position information indicates the position of a node (corresponding to a category keyword) in the tree-structured graph, and can be expressed in, for example, a tuple format. In the tuple-formatted position information, each number in the order of a numeric sequence indicates the position of a node to be traced for each layer in the tree-structured graph.

[0021] For example, suppose the position information for a node is (0,2,25,11,2). Each number indicates the position of the child node from the left in the parent node, starting from the top layer. The leftmost value in the position information is '0', as it indicates the topmost (first layer) node (root node). The second number in the position information, '2', indicates the second child node from the left of the parent node, one layer below the root node (second layer). For example, in FIG. 5, the location information of the category keyword "Japanese politicians" of id1 is (0), and the location information of the category keyword "Tokyo politicians" of id3 is (0,1,1). The category layering means 100 outputs the generated category layer information (ID and location information) to the category information extraction means 102.

[0022] The person KW extraction means (person keyword extraction means) 101 extracts keywords of person names (person KWs) from the dataset. Here, the person KW extraction means 101 extracts keywords corresponding to categories including predetermined character strings that identify people from the dataset as person KWs. Specifically, the person KW extraction means 101 searches for keywords containing predetermined words that can be used to identify a person, such as "year of birth," "year of death," and "living person," and extracts keywords that contain the ID corresponding to the keyword as a category ID as person KW.

[0023] For example, the person KW extraction means 101 searches for IDs containing the keyword phrase "grade" in the keyword table Tk of Fig. 2. Then, the person KW extraction means 101 further searches for IDs that contain the searched ID (id8 in this case) as a category ID in the category table Tc of Fig. 3. Then, the person KW extraction means 101 extracts "Taro X" corresponding to the searched ID (id6 in this case) as a person KW in the keyword table Tk of Fig. 2. In addition, Wikipedia's process of identifying keywords that include the categories "year of birth," "year of death," or "living person" as person KWs has been experimentally confirmed to be 100% accurate.

[0024] Of course, keywords for people in a data set may also be keywords that include a phrase that identifies a person in the person's name. For example, a keyword that includes the predetermined phrase "person" such as "Taro X (person)" may be used. In this case, the person KW extraction means 101 may extract keywords that include the phrase "person" from the keyword table Tk as person KWs. The person KW extraction means 101 outputs the extracted person KW to the category information extraction means 102 and the face image collection means 11.

[0025] The category information extraction means 102 extracts, as category information, hierarchical categories corresponding to keywords of person names from the data set based on the category hierarchical information hierarchized by the category hierarchization means 100 . The category information extraction means 102 uses the category ID corresponding to the person KW extracted by the person KW extraction means 101 as a reference and extracts higher-level IDs in the tree structure from the category hierarchy information. Then, the category information extraction means 102 generates keywords corresponding to the reference category ID and the higher-level category ID as category information.

[0026] For example, even if only "id5" is set as the category ID for a certain person KW "△△Hanako" in the category table Tc (Figure 3), the category information extraction means 102 generates, as category information, keywords corresponding to the ID "id5" and the IDs "id1, id2, id3, id4" that are hierarchically ranked higher in the category hierarchy information, as shown in Figure 6. This allows the category information extraction means 102 to associate not only category keywords directly corresponding to person keywords but also category keywords higher in the hierarchical structure with person keywords. The category information extraction means 102 associates the person KW with the category information and outputs it to the data integration means 13.

[0027] Note that tree-structured position information may be added to each keyword in the category information, which makes it possible to create a hierarchical database of face recognition training data.

[0028] The face image collecting means 11 collects face images corresponding to person KW. The facial image collection means 11 includes a web search means 110 and a facial image extraction means 111.

[0029] The Web search means 110 searches for and collects documents (Web documents) related to keywords of people's names on the Web, which is a system for publishing and viewing documents on a network (Internet) N. The web search means 110 uses a general search engine via the network N to search and collect web documents related to the person keywords extracted by the person keyword extraction means 101 of the data analysis means 10 from the web server S. Note that the search for person keywords using a search engine can be performed using general web scraping technology. The Web documents collected by the Web search means 110 include images including the faces of the person KW, images other than the faces, text data, and the like. The web search means 110 outputs the collected web documents to the face image extraction means 111 .

[0030] The facial image extraction means 111 extracts facial images from images contained in Web documents collected by the Web search means 110 by facial recognition. The facial image extraction means 111 detects faces in images contained in Web documents by general facial recognition processing, and extracts only images that show one face. The facial image extraction means 111 outputs the extracted facial image and the Web document from which the facial image was extracted, together with the person keyword, to the person keyword correspondence determination means 12.

[0031] The person keyword correspondence determination means (person keyword correspondence determination means) 12 determines whether or not the face image collected by the face image collection means 11 is a face image of a person of the person keyword. The person keyword correspondence determination means 12 includes a feature amount extraction means 120 , a clustering means 121 , a relevance setting means 122 , and a determination means 123 .

[0032] The feature extraction means 120 extracts features (facial feature amounts) from the face images collected by the face image collection means 11. For example, the feature extraction means 120 extracts facial feature F by performing multiple convolution operations on a facial image I using a pre-trained convolutional neural network NN equipped with multiple convolution layers CL (CL1, CL2, ...) as shown in Fig. 7. This facial feature F has a dimension of, for example, about 500. This convolutional neural network NN can use, for example, a network for the feature extraction part of a general object recognition network such as VGG (Visual Geometry Group) 19. The feature extraction means 120 extracts facial features {f1, f2, ..., f n} (n is the number of face images) for each person keyword to the clustering means 121.

[0033] The clustering means 121 clusters the facial feature amounts extracted by the feature amount extraction means 120 . A general method may be used as the clustering method in the clustering means 121. For example, the clustering means 121 uses a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, which can also calculate the number of classes, to calculate the facial feature values ​​{f1, f2, ..., f n} are clustered.

[0034] Fig. 8 shows the concept of facial features clustered by the clustering means 121. Fig. 8 shows a state in which facial features {f1, f2, ..., f9} are classified into three classes (C1, C2, C3), with facial features {f1, f4, f5, f9} classified into class C1, facial features {f6, f8} classified into class C2, and facial features {f2, f3, f7} classified into class C3. This allows the clustering means 121 to divide a plurality of facial features into a plurality of groups, such as a group of people corresponding to the person KW and other groups, or even if people correspond to the same person KW, into groups such as faces facing forward and faces facing sideways. The clustering means 121 outputs the clustering result to the determination means 123 .

[0035] The relevance setting means 122 sets a relevance indicating the degree of relevance between the facial feature amount and a person's name in association with the facial feature amount based on a Web document including a face image from which the facial feature amount has been extracted. The relevance setting means 122 sets the relevance based on a predetermined criterion. Here, the relevance setting means 122 increases the relevance based on the criterion from a predetermined initial value (for example, "0").

[0036] For example, if the Web document from which the face image has been extracted is a Wikipedia document that uses a person keyword as an entry term, the relevance setting means 122 adds a predetermined number to the relevance. Furthermore, for example, when a tag indicating an image in a Web document contains a description of a person keyword, the relevance setting means 122 adds a predetermined number to the relevance. Furthermore, for example, when a Web document is similar to a description that explains a person keyword included in the dataset, the relevance setting means 122 adds a predetermined number to the relevance. This similarity can be determined, for example, when the number of words common to the Web document and the description is greater than a predetermined number, or when the ratio of the number of common words to the total number of words in the Web document and the description is greater than a predetermined ratio. Furthermore, if the web documents are categorized in advance, such as in Wikipedia, it is possible to update the relevance by calculating the degree of overlap with the category information of the person keyword and adding it to the relevance. In the above relevance calculation, the elements for calculating relevance, such as words in a sentence or image features, can be converted into vectors in a common dimensional space using WORD2VEC, PCA (Principal Component Analysis), etc., and then similarity calculation processing such as cosine similarity can be performed to obtain the relevance. The relevance setting means 122 sets relevance W{w1, w2, . . . , w} for a plurality of face features corresponding to a person keyword. n} (n is the number of face images) for each person KW to the determination means 123.

[0037] The determination means 123 determines whether or not each class corresponds to a person's name based on the degree of association set for the facial feature amounts belonging to each class clustered by the clustering means 121. This determination means 123 calculates the sum of the relevance set by the relevance setting means 122 for the facial features belonging to each clustered class, and determines that a class whose sum satisfies a predetermined criterion is the class corresponding to the person keyword.

[0038] For example, suppose the clustering result for a certain person keyword is as shown in Fig. 8. Also, suppose the relevance of the person keyword set to the facial feature {f1, f2, ..., f9} is {w1, w2, ..., w9}. In this case, the determination means 123 calculates the sum W1 of the relevance of class C1 as w1 + w4 + w5 + w9, the sum W2 of the relevance of class C2 as w6 + w8, and the sum W3 of the relevance of class C3 as w2 + w3 + w7.

[0039] Furthermore, the determination means 123 determines whether each class corresponds to a person keyword based on, for example, whether the sum of the degrees of association for each class exceeds a predetermined threshold, or whether the difference between the sum of the degrees of association for each class and the sums of other classes is greater than a predetermined threshold. The determination means 123 associates the person KW with the face feature determined to be the class of the person KW, and outputs the result to the data integration means 13.

[0040] The data integration means 13 integrates category information corresponding to the person keywords generated by the data analysis means 10 with facial features determined to correspond to the person keywords by the person keyword correspondence determination means 12, to generate learning data for face recognition. The data integration means 13 generates face recognition learning data by linking the facial feature amount F and the category information CI for each person KW, as shown in Fig. 9. Note that tree-structured position information may be added to each keyword in the category information CI. Furthermore, the data integration means 13 may add a face image from which facial features have been extracted to the face recognition learning data in place of or together with the facial features.

[0041] By configuring as described above, the face recognition learning data generation device 1 can collect face images and generate learning data for face recognition with person keywords and category information added without any manual effort. Furthermore, the face recognition training data generation device 1 can collect face images in a wide variety of categories, as well as in hierarchical categories, thereby enabling the face recognition training data generation device 1 to generate comprehensive and systematically organized training data. The face recognition learning data generation device 1 can be operated by a program (face recognition learning data generation program) that causes a computer to function as each of the above-mentioned means.

[0042] [Operation of the face recognition learning data generation device] Next, with reference to FIG. 10 (for the configuration, refer to FIG. 1 as appropriate), the operation of the face recognition learning data generation device 1 according to the embodiment of the present invention will be described. In step S1, the category layering means 100 of the data analysis means 10 analyzes the data set and layers the categories. Here, the category layering means 100 refers to a hierarchy table Tu (FIG. 4), sequentially and recursively associates a higher-level hierarchy ID with each ID (identifier), thereby layering the categories and generating category hierarchy information (FIG. 5) in which the IDs and position information are associated with the nodes of a tree-structure graph.

[0043] In step S2, the person KW extraction means 101 of the data analysis means 10 analyzes the dataset and extracts person KWs from the keywords of the dataset. Here, the person KW extraction means 101 extracts, as person KWs, keywords in the keyword table Tk (FIG. 2) whose category table Tc (FIG. 3) contains an ID including any of the phrases "year of birth," "year of death," and "living person."

[0044] In step S3, the category information extraction means 102 of the data analysis means 10 extracts category KWs corresponding to the person KWs extracted in step S2 from the dataset based on the category hierarchy information generated in step S1, and generates category information of the person KWs (Figure 6).

[0045] In step S4, the web search means 110 of the face image collection means 11 collects web documents corresponding to the person keywords extracted in step S2 from the web server S via the network N. Here, the web search means 110 collects web documents by web scraping using the person keywords.

[0046] In step S5, the facial image extraction means 111 of the facial image collection means 11 extracts facial images from the images included in the Web documents collected in step S4. Here, the facial image extraction means 111 detects faces from the Web documents by facial recognition and extracts only images that show one face.

[0047] In step S6, the feature extraction means 120 of the person keyword correspondence determination means 12 extracts features (facial features) from the face image extracted in step S5. Here, the feature extraction means 120 extracts facial features from the face image using a pre-trained convolutional neural network. In step S7, the clustering means 121 of the person keyword correspondence determination means 12 clusters the facial feature amounts extracted in step S6.

[0048] In step S8, the relevance setting means 122 of the person KW correspondence determination means 12 sets the relevance between the facial feature extracted in step S6 and the person KW. For example, when the Web document is a Wikipedia document that uses the person KW as a headword, the relevance setting means 122 sets a larger relevance value as the relevance between the Web document and the person KW increases.

[0049] In step S9, the determination means 123 of the person keyword correspondence determination means 12 determines, for each class clustered in step S7, whether or not the class corresponds to a person keyword based on the relevance set in step S8. Here, the determination means 123 calculates the sum of the relevance degrees for the facial features included in the class, and determines that a class whose sum of the relevance degrees is greater than a predetermined threshold is a class corresponding to a person keyword.

[0050] In step S10, the data integration means 13 integrates, for each person keyword, the category information generated in step S3 corresponding to the person keyword and the facial features belonging to the class determined in step S9 to be the class of the person keyword, to generate learning data for face recognition (FIG. 9). Through the above operations, the face recognition learning data generation device 1 can automatically collect face images corresponding to person KWs and generate face recognition learning data.

[0051] [Configuration of face recognition device] Next, the configuration of the face recognition device 2 according to an embodiment of the present invention will be described with reference to FIG. The face recognition device 2 recognizes the faces of people captured in the video. As shown in FIG. 11, the face recognition device 2 includes a face recognition learning data storage means 20, a face detection means 21, a feature extraction means 22, a similarity calculation means 23, and a person identification means 24.

[0052] The face recognition learning data storage means 20 stores face recognition learning data, and can be configured as a general storage medium such as a hard disk. The face recognition learning data storage means 20 stores in advance the face recognition learning data generated by the face recognition learning data generation device 1 described with reference to FIG. As shown in FIG. 9, the face recognition learning data is a database in which person KW is associated with face feature amounts F and category information CI.

[0053] The face detection means 21 detects faces appearing in the input video image. The face detection means 21 recognizes faces in the video image by using a general face recognition process. The face detection means 21 extracts the detected face area (face image) from the video for each frame and outputs it to the feature extraction means 22.

[0054] The feature extraction means 22 extracts facial features from the face image detected by the face detection means 21. This feature extraction means 22 extracts facial features using the same method as the feature extraction means 120 of the face recognition training data generation device 1 described in FIG. The feature extraction means 22 outputs the extracted facial feature amount or amounts to the similarity calculation means 23.

[0055] The similarity calculation means 23 calculates the degree of similarity (similarity) between the facial feature amount extracted by the feature amount extraction means 22 and the facial feature amount stored in the face recognition learning data storage means 20. Here, the similarity calculation means 23 calculates the similarity for the facial feature amounts that correspond to the category information (keywords) specified externally, among the facial feature amounts stored in the face recognition learning data storage means 20. For example, if the keyword "politicians in Tokyo" is specified as category information, the similarity calculation means 23 will calculate the similarity only for the facial features of the learning data for face recognition stored in the face recognition learning data storage means 20, whose category information includes the keyword "politicians in Tokyo."

[0056] The similarity calculation method used by the similarity calculation means 23 may be a general method, and for example, the similarity may be calculated using the cosine similarity between the vector of facial features extracted by the feature extraction means 22 and the vector of facial features stored in the face recognition learning data storage means 20. The similarity calculation means 23 outputs the calculated similarity and the person KW of the face recognition learning data used to calculate the similarity to the person identification means 24.

[0057] The person identification means 24 identifies a person based on the similarity calculated by the similarity calculation means 23. The person identification means 24 extracts those whose similarities calculated by the similarity calculation means 23 are greater than a predetermined threshold.The person identification means 24 then sorts the extracted similarities and outputs the person keywords of the face recognition learning data used to calculate the similarities as recognition results in descending order of similarity.Of course, for simplicity, it is also possible to output only the person keyword with the highest similarity.

[0058] In addition, the person identification means 24 may output keywords of the specified category information in association with the person KW, or keywords of all category information associated with the person KW stored in the face recognition learning data storage means 20. This makes it possible to add metadata based on category information to a person keyword recognized by the face recognition device 2. In addition, if tree-structured location information is added to the category information of the face recognition learning data stored in the face recognition learning data storage means 20, the person identification means 24 may output the location information in addition to the category information in association with the person KW.

[0059] With the configuration described above, the face recognition device 2 can perform face recognition by specifying category information, thereby improving the accuracy of face recognition. The face recognition device 2 can be operated by a program (face recognition program) that causes a computer to function as each of the above-mentioned means.

[0060] [Facial recognition device in action] Next, the operation of the face recognition device 2 according to an embodiment of the present invention will be described with reference to Fig. 12 (for the configuration, see Fig. 11 as appropriate). It is assumed that the face recognition learning data storage means 20 stores in advance the face recognition learning data generated by the face recognition learning data generation device 1 (Fig. 1).

[0061] In step S20, the face detection means 21 detects a face appearing in the input video, and extracts the area of ​​the detected face as a face image. In step S21, the feature extraction means 22 extracts features (face feature amounts) from the face area (face image) detected in step S20.

[0062] In step S22, the similarity calculation means 23 calculates the degree of similarity (similarity) between the facial features extracted in step S21 and the facial features stored in the face recognition learning data storage means 20 and corresponding to the externally specified category information (keywords). In step S23, the person identification means 24 identifies a person based on the similarity calculated in step S22. At this time, the person identification means 24 extracts and sorts person keywords whose similarity is greater than a predetermined threshold, and outputs them as recognition results in descending order of similarity. Through the above operations, the face recognition device 2 can recognize faces from video by limiting the categories.

[0063] Although the embodiment of the present invention has been described above, the present invention is not limited to this embodiment. In this embodiment, the dataset contains keywords including person names, and the target person is identified using the person name keyword. However, the keyword does not have to be a person's name as long as it can identify the target person. For example, the keyword may be a number representing a person. For example, it may be an index such as a My Number that can identify a person. Furthermore, the text of the dataset itself may have person names replaced with other indexes so that individuals cannot be identified.

[0064] In this embodiment, documents containing facial images are described as being searched and collected on the Web. However, documents containing facial images are not limited to those searched and collected on the Web, and can be anything that contains a facial image and document data related to that face. For example, documents containing facial images can be documents stored in advance on a storage medium, or documents can be loaded from an external source. Therefore, the Web search means 110 is not limited to searching on the Web, and can also be configured as a collection means. Note that the documents in this case are multimedia documents containing text, images, and audio.

[0065] In this embodiment, the determination means 123 calculates the sum of the relevance for each clustered class and performs the determination based on the calculated value. However, the index used for the determination is not limited to the sum of the relevance for each class. The index used for the determination may be any index that can be determined to correspond to a person keyword. For example, the index may be the average relevance of the class, the maximum value of the facial feature amount belonging to the class, or the like. [Explanation of symbols]

[0066] 1. Device for generating learning data for face recognition 10 Data analysis methods 100 Category Hierarchy Method 101 Person KW extraction method 102 Category information extraction method 11 Facial image collection method 110 Web search methods (collection methods) 111 Facial image extraction means 12 Person keyword correspondence determination method 120 Feature Extraction Method 121 Clustering Methods 122 Means of setting relevance 123 Judgment means 13 Data Integration Methods 2. Facial Recognition Device 20. Face recognition learning data storage means 21 Face detection means 22 Feature extraction method (Second feature extraction means) 23. Similarity Calculation Methods 24. Character-Specific Methods

Claims

1. a person keyword extraction means for extracting keywords of person names from a data set in which keywords are associated with hierarchical categories; and a category information extraction means for extracting, as category information, the hierarchical categories corresponding to the keywords that identify people from the data set; a collection means for collecting documents related to a keyword that identifies the person; and a face image extraction means for extracting a face image from an image included in the document by face recognition; a feature extraction means for extracting features from the face image; a relevance setting means for setting a relevance between the feature amount and the person specified by the keyword in association with the feature amount based on the document including the face image from which the feature amount has been extracted; clustering means for clustering the feature quantities of the face image; a determination means for determining whether each of the clustered classes corresponds to the specified person based on the relevance set for the feature amounts belonging to each of the clustered classes; a data integration means for integrating, for each keyword that identifies the person, features belonging to a class determined to correspond to the identified person with the category information to generate face recognition learning data; a face recognition learning data storage means for storing the face recognition learning data; a face detection means for detecting a face in the video and extracting a face image; a second feature extraction means for extracting features from the face image extracted by the face detection means; a similarity calculation means for calculating a similarity between a feature of a face image corresponding to designated category information in the face recognition learning data and the feature extracted by the second feature extraction means; a person identification means for identifying a person name corresponding to a feature value having a similarity greater than a predetermined threshold value as a recognition result; A face recognition device comprising:

2. A face recognition program for causing a computer to function as the face recognition device according to claim 1.

Images